The following problem and accompanying solution were posted to sage-devel. I have polished them up a bit and put them here so they won’t be buried in the huge sage-devel mailing list.
- Build a Sage source distribution.
- Apply your patches to the Sage library.
- Produce a new source tarball based on your patched Sage source distribution.
- Compile your newly wrapped up modified Sage source tarball.
The key problem is that you don’t want to run through two separate compilation processes.
The following solution uses the queues extension of Mercurial. In a Sage source tarball, the Sage library is wrapped up as the package
Uncompress that bzip2 compressed tarball to get a directory named
If you want, you could delete
Now patch the Sage library as if it had been built and found under SAGE_ROOT/devel:
$ cd SAGE_ROOT/spkg/standard/sage-x.y.z/ $ hg qimport /URL/or/path/to/patch.patch $ hg qpush $ <repeat-previous-two-commands-as-often-as-required>
$ hg qapplied
should return a list of patches you have applied using “hg qpush“. Once you are happy that you have applied all necessary patches, wrap up the patched Sage library:
$ pwd SAGE_ROOT/spkg/standard/sage-x.y.z/ $ hg qfinish -a $ cd ..
Now you should have
so you could delete
Finally, proceed to build Sage from source.
This is a revision of my post to the sage-release mailing list, outlining some guidelines for merging positively reviewed tickets. I want to post it here so that it won’t be lost in the archive of the sage-release mailing list.
There is as yet little to no guidelines for Sage release management. The release management wiki page has been around for months, but it’s not especially helpful for anyone who wants hands-on experience. Here, I’ll try to list some guidelines for merging tickets/spkg’s and managing positively reviewed tickets. The primary focus is on merging tickets with patches.
Before starting on merging tickets for a new milestone, I would compile the source tarball of the current stable release of Sage under a fast-ish directory such as /scratch/. In my case, it’s under my /scratch/ home directory /scratch/mvngu/release/ on the machine sage.math. The idea is to avoid working under a directory that resides on an NFS mounted disk. Towards that end, I also set my default DOT_SAGE directory to a fast-ish directory, as implemented via the following export line in my .bashrc:
# The Sage working directory .sage/ export DOT_SAGE=/dev/shm/mvngu/dot_sage
It also helps to know whether or not there’s any doctest failure on sage.math with the newly compiled source tarball.
After compiling and (parallel) doctesting the latest version of Sage, it’s time to merge tickets that have received positive review. I often use trac report 11 to find out which tickets have been positively reviewed. Using report 11, I would then pick a ticket, read through it for any special instructions. In many cases, the ticket has instructions for which patch to merge and, for more than one patch on the ticket, in which order. In some cases, it can be difficult to work out the order in which to merge patches on a ticket. With a situation like this, I would request the patch author(s) to explain how to merge patches on the ticket and in which order.
Sage has a merge script in the scripts repository. This script is designed for merging tickets. The merge script is SAGE_LOCAL/bin/sage-apply-ticket, which can be used from the command line interface via
./sage -merge <ticket-number>
However, to effectively use that script, one first needs a working knowledge of Mercurial and its queues extension. This is especially important because the merge script uses Mercurial queues (mq). There would be times when the merge script cannot help me with debugging ticket conflicts, so I would need to use mq together with the bisect command to search through the commit history. I don’t often use the merge script, so there is little for me to say about how to effectively use it.
Here, I’ll explain one way to merge a positively reviewed ticket using mq. Ensure that any patches to be merged must have the full name and email address of the patch author. The commit message must contain the ticket number. Use mq to qimport and qpush any relevant patches to be merged. Sometimes a patch author would forget to put in their full name and email address on the patch, so I need to manually download the patch and fill in those details. Another way is to set the username field in my .hgrc to the patch author’s full name and email address; then qimport and qpush. After that, I would restore my full name and email address in my .hgrc. If a commit message doesn’t contain the ticket number, or the message is something non-sensical like
I would use qrefresh -e to replace that with a sensible commit message that also has the ticket number. Now the patches must pass the following criteria:
- No hunk failures resulting from qpush.
- The Sage library must rebuild without errors.
- The HTML and PDF versions of the reference manual must build successfully.
- No doctest failures when (parallel) doctesting everything under
If any of the above steps results in an error, then it’s very likely that the ticket would receive an automatic “needs work” status. Be sure to explain on the ticket why the patch(es) need work. Is it a rebase? If so, which version of Sage to perform the rebase on. Does the ticket conflict with another ticket? And so on.
If all of the above steps pass for the ticket, then move on to another ticket for merging. Be careful not to close the ticket even though the above criteria are met. What I would do is, merge a dozen or so tickets as per the above instructions, but not closing the corresponding tickets after qimport and qpush their patches. Once I have a qseries with patches from a dozen tickets, which should still be open by now, I would copy everything under SAGE_ROOT to another directory, say, under /dev/shm/. (I would also note down the order in which those dozen tickets have been merged.) From that newly copied SAGE_ROOT, I would create a source tarball to test that everything still builds after merging the dozen tickets. To do so, I would do:
- cd to the SAGE_ROOT under /dev/shm/.
- Run ./sage.
- Run ./sage -b main. This step would take a while, so I would do something during the rebuild process that would occupy me for at least 15 minutes.
- Create a tentative source tarball with
./sage -sdist x.y.z-<descriptive-name>
The name <descriptive-name> is something to remind myself that this source tarball is only to test that the resulting tarball still builds and all doctests still pass.
- The result is a source tarball under dist/.
- Untar that source tarball, compile Sage, and parallel doctest. This step usually takes more than 1 hour, so I would make sure to have something to occupy me for at least an hour.
To be safe, I often copy the whole SAGE_ROOT of my merge tree to another directory and run through the above steps. This is because a ticket can have an spkg for merging into the standard spkg repository. You could push a sage-main repository to another copy of Sage somewhere, but presently you can’t do the same thing for the standard spkg repository.
Once the tentative source tarball successfully compiles and all doctests under SAGE_ROOT/devel/sage-main/ pass, then I would close the relevant dozen or so tickets.
The above is a general overview of the process I follow for merging/closing tickets. It’s a slow process, I admit. But I would rather try to catch build and doctest problems early, than have them bit me later on down the track. As noted at the beginning of this post, I didn’t cover anything about merging updated spkg’s in the standard packages repository. That could be a topic for another post.
An email on sage-devel asks about how to use Mercurial for managing patches. I replied that tickets #8108 and #8147 have some introductory materials on that topic. However, I didn’t pick up the irony in my reply, hence the following insightful remark:
To learn more about patch management, please apply the following