Home > bug fix, documentation, Sage > How to patch a Sage package

How to patch a Sage package

What is an spkg?

The abbreviation “spkg” stands for “Sage package”. The directory SAGE_ROOT/spkg/standard contains spkg’s. In a source install, these are all Sage spkg files (actually .tar or .tar.bz2 files), which are the source code that defines Sage. In a binary install some of these may be small placeholder files to save space.

Sage packages are distributed as .spkg files, which are .tar.bz2 files (or tar files) but have the extension .spkg to discourage confusion. Although Sage packages are packed using tar and/or bzip2, please note that .spkg files contain control information (installation scripts and metadata) that are necessary for building and installing them. For source distributions, when you compile Sage the file SAGE_ROOT/makefile takes care of the unpacking, compilation, and installation of Sage packages for you. For more information on the structure of .spkg files, please refer to the Sage Developer’s Guide in your local installation of Sage at

SAGE_ROOT/sage/doc/output/html/en/developer/index.html

If you cannot locate that file in your local installation of Sage, you might want to consider (re)building the standard Sage documentation using this command:

SAGE_ROOT/sage -docbuild all html

or see the Developers’ guide, especially the section “Producing New Sage Packages”. Additional Sage packages can be found on the Sage website.

How do I patch an spkg?

Make sure you are familiar with the structure and conventions relating to spkg’s; see the section Producing New Sage Packages for details. Patching an spkg involves patching the installation script of the spkg and/or patching the upstream source code contained in the spkg. Say you want to patch the GAP package gap-4.4.10.p17. Note that “p17” denotes the patch level of the spkg. The installation script of that spkg is

gap-4.4.10.p17/spkg-install

In general, a script with the name spkg-install is an installation script for an spkg. To patch the installation script, use a text editor to edit that script. Then in the log file SPKG.txt, provide a high-level description of your changes. Once you are satisfied with your changes in the installation script and the log file SPKG.txt, use Mercurial to check in your changes and make sure to provide a meaningful commit message.

The directory src/ contains the source code provided by the upstream project. For example, the source code of GAP 4.4.10 is contained under

gap-4.4.10.p17/src/

To patch the upstream source code, you should first copy the relevant file under src/ over to the directory patches/. Then edit that copied file rather than the relevant file under src/. Once you are satisfied with your changes to the copied file under patches/, generate a unified diff between the original file under src/ and the corresponding copied (and edited) file under patches/. Save the unified diff to a file with the same name, but use the file extension “.patch”, and place the diff file under patches/. Note that both of the directories patches/ and src/ should not be under revision control. The Mercurial configuration file .hgignore should contain the following two lines

patches/
src/

Make sure that the installation script spkg-install contains code to copy over the patched file under patches/ to the relevant place under src/. For example, the file

gap-4.4.10.p17/patches/configure.out-ia64

is a patched version of the file

gap-4.4.10.p17/src/cnf/configure.out

The installation script gap-4.4.10.p17/spkg-install contains the following conditional to test under which conditions to copy over the patched file to the relevant place under src/:

# If we're on an Itanium Linux box, we overwite configure.out 
# with a slightly modified version.  The modified version has
# all -O2's replaced by -O0. See 
# http://www.gap-system.org/Faq/Hardware-OS/hardware-os8.html
# On the San Diego Super computer `uname -p` is unknown, but
# uname -a includes ia64 in the output.  So this is a better
# detection method. Note that GAP was "fixed" to work fine on
# Itanium without this -O0 hack, but with GCC-4.4.0, GAP 
# mysterious stopped working again.  So we revert to using -O0.
if [ `uname` = "Linux" ]; then
   uname -a |grep ia64
   if [ $? = 0 -o `uname -p` = "ia64" ]; then
       cp patches/configure.out-ia64 src/cnf/configure.out
       echo "The file configure.out was patched for SAGE!" > src/cnf/configure.out.README
   fi
fi

Now provide a high-level explanation of your changes in SPKG.txt. Once you are satisfied with your changes, use Mercurial to check in your changes with a meaningful commit message. Next, increment the patch level of the spkg by one, e.g. rename gap-4.4.10.p17 to gap-4.4.10.p18. If you want your patched spkg to be reviewed and available in the next release of Sage, you need to compress it using tar and bz2. Under Linux and OS X, you can compress your patched spkg as follows:

tar -jcf gap-4.4.10.p18.spkg gap-4.4.10.p18

As part of your request for your patched spkg to be reviewed, you need to provide a URL to your spkg on the relevant trac ticket and/or on an email to the relevant mailing list. Usually, you should not upload your spkg to the relevant trac ticket.

Advertisements
  1. malb
    20 January 2010 at 10:22 pm

    Minh, there is a special command for compressing SPKGs which checks a few simple things:

    sage -spkg foobar/

  2. 20 January 2010 at 10:35 pm

    @malb The command line options “-pkg” and “-pkg_nc” to Sage are useful when you already have Sage installed on your system. For systems that don’t have Sage installed, tar and bz2 are more “portable” 🙂 Thanks for the clarification.

  3. 21 January 2010 at 3:00 am

    what is still unclear here, is how to bump up versions of packages.

    Say, I want to replace gap-4.4.10 with gap-4.4.12; so this not exactly patching
    of gap-4.4.10, but a complete replacement.
    (say, I want to install a version made by Dave Joyner some time ago)

    If I try something naive with this, I kind of confuse Sage, that still thinks
    that the GAP package version ends with 10, not with 12.

    Are there any “hooks” in Sage code outside the spkg that hold the version
    information? (if yes, it would be a nightmare…)

    Thanks,
    Dima

  4. 21 January 2010 at 6:44 am

    > what is still unclear here, is how to bump up versions of packages.

    If you want to bump up the version of an spkg, you need to follow some naming conventions. Use the name and version number as given by the upstream project, e.g. “gap-4.4.12”. If the upstream package is taken from some revision other than the stable version, you need to append the date at which the revision is made, e.g. the Singular package “singular-3-1-0-4-20090818.p3.spkg” is made with the revision as of 2009-08-18. If you start afresh from an upstream release without any patches to its source code, the resulting spkg need not have any patch-level labels. For example, sagenb-0.6.spkg is taken from the upstream stable version sagenb-0.6 without any patches applied to its source code. So you don’t see any patch-level numbering such as “.p0” or “.p1”.

    > Say, I want to replace gap-4.4.10 with gap-4.4.12; so this not
    > exactly patching of gap-4.4.10, but a complete replacement.

    Follow the naming conventions as described above for your newer upstream release version. If necessary, apply any patches to that upstream version and put in patch-level numbering. Then package your replacement spkg using tar and bz2, or the command “sage -pkg package-x.y.z”. The command “sage -pkg package-x.y.z” assumes that you already have Sage installed somewhere that you could use. It automatically appends the extension “.spkg” to the resulting (compressed) spkg. However, using tar and bz2, you need to manually put in the extension “.spg” instead of “.tar.bz2”. Also note that you could package up your replacement spkg using “sage -pkg_nc “. This command doesn’t use any compression; it usually produces a tar archive of the given directory name. You use this command if your spkg has many binaries.

    Then to install your replacement spkg, you use “sage -f /URL/to/package-x.y.z.spkg”. To compile Sage from source with the replacement (standard) spkg, replace the existing spkg under SAGE_ROOT/spkg/standard. In its place, put your replacement spkg. Then execute “make” from SAGE_ROOT.

    > If I try something naive with this, I kind of confuse Sage,
    > that still thinks that the GAP package version ends with 10,
    > not with 12.

    If the above process doesn’t work, I like to know.

    > Are there any “hooks” in Sage code outside the spkg that hold the version
    > information? (if yes, it would be a nightmare…)

    I’m not aware of any file or code that holds the version information. Note that the file SAGE_ROOT/devel/sage-main/sage/version.py contains version information for the current Sage installation. However, that file is automatically generated for each release by the file SAGE_LOCAL/bin/sage-sdist. The script “SAGE_ROOT/spkg/standard/newest_version” deals with newer releases of spkg’s, and the script “SAGE_ROOT/spkg/install” deals with the installation of spkg’s. But they don’t and should not hardcode version numbers.

  5. 25 January 2010 at 11:28 am

    you described how one makes a “conditional” patch. I rather need to have an
    unconditional patch: something that changes GAP_ROOT/src/saveload.c file,
    no matter what.
    Do I still have to put the changed file in subdirectory patches/ ?

  6. 25 January 2010 at 11:40 am

    where should the .hgignore reside? within spkg, at the top level?

    what is the purpose of a binary file “patch” at the top level of spkg?
    is it just a rubbish leftover?

  7. 25 January 2010 at 11:55 am

    > I rather need to have an unconditional patch: something that
    > changes GAP_ROOT/src/saveload.c file, no matter what.

    @Dima — As you pointed out, a conditional patch depends on a conditional holding true. For example, is the platform Cygwin, FreeBSD, Linux, Mac OS X, Unix, Windows? And so on. For an unconditional patch, you don’t need to surround the copy command with a conditional. But immediately after the copy line, you need to test that the copy process is successful. For example, here’s a block in “spkg-install” from python-2.6.4.p5 that copies over an unconditional patch:

    cp patches/locale.py src/Lib/locale.py
    if [ $? -ne 0 ]; then
        echo "Error copying patched locale.py"
        exit 1
    fi
    

    > Do I still have to put the changed file in subdirectory patches/ ?

    Put the changed file under “patches/” and leave the original file where it is under “src/”, even if you are making an unconditional patch. The idea is to have a line in “spkg-install” that copies the patched file from “patches/” over to the relevant place under “src/”.

    > where should the .hgignore reside? within spkg, at the top level?

    The file “.hgignore” should be where “spkg-install”, “patches/”, “src/” and “SPKG.txt” are. For example,

    [mvngu@sage python-2.6.4.p5]$ ls -A
    .hg  .hgignore  patches  spkg-install  SPKG.txt  src
    

    > what is the purpose of a binary file “patch” at the top level of spkg?
    > is it just a rubbish leftover?

    I’m not sure I understand you. Could you give me an example spkg?

  8. 25 January 2010 at 3:04 pm

    I wonder how do I *reinstall* the same package again.
    Somehow, it looks as if I managed to corrupt the installation.
    It started, after a couple of errors, complaining of the size mismatch in GAP workspaces, even though I remove the old GAP workspaces…

  9. 25 January 2010 at 3:10 pm

    > I wonder how do I *reinstall* the same package again.

    The command “sage -i package.spkg” installs a package. The command “sage -f package.spkg” *forces* a (re)installation of a package. You can use “sage -f package.spkg” to reinstall a package.

    > Somehow, it looks as if I managed to corrupt the installation.
    > It started, after a couple of errors, complaining of the size
    > mismatch in GAP workspaces, even though I remove the old GAP
    > workspaces…

    Can you email sage-devel about this and provide a link to your package?

  10. 25 January 2010 at 5:48 pm

    never mind. I did something stupid to GAP’s garbage collector, and the hell broke loose. Now it seems all be working, finally…

  11. 26 January 2010 at 10:54 am

    say, I have a new spkg without .hg/, yet.
    How exactly do I put it under hg (using hg_sage, or???) ?
    (It’d not be a new project, but a part of the existing Sage “tree”, right?)

  12. 28 January 2010 at 3:08 am

    > say, I have a new spkg without .hg/, yet.
    > How exactly do I put it under hg (using hg_sage, or???) ?
    > (It’d not be a new project, but a part of the existing Sage “tree”, right?)

    @Dima
    For a given spkg, the directory src/ contains the source code of the upstream project. If you want to upgrade the current source code under src/ with a newer version, you could delete the src/ directory, uncompress the newer upstream tarball, rename that uncompressed tarball to src/ and move src/ to be under the spkg directory. For example, say you start with gap-4.4.10.p13 which contains GAP 4.4.10. Now you want to upgrade that spkg to GAP 4.4.12, which is contained in the tarball gap4r4p12.tar.bz2. For this example, assume that you have this directory listing:

    [mvngu@sage gap]$ ls
    gap-4.4.10.p13  gap-4.4.10.p13.spkg  gap4r4  gap4r4p12.tar.bz2
    

    Now rename gap-4.4.10.p13 to gap-4.4.12, remove the directory gap-4.4.12/src/, rename gap4r4 to src/, and put src/ under gap-4.4.12/:

    [mvngu@sage gap]$ mv gap-4.4.10.p13 gap-4.4.12
    [mvngu@sage gap]$ rm -rf gap-4.4.12/src/
    [mvngu@sage gap]$ mv gap4r4 src
    [mvngu@sage gap]$ mv src/ gap-4.4.12/
    [mvngu@sage gap]$ ls
    gap-4.4.10.p13.spkg  gap-4.4.12  gap4r4p12.tar.bz2
    [mvngu@sage gap]$ head gap-4.4.12//src/README 
                      README for GAP 4.4.12
    
    These archives contain the current release of GAP. It is version
    4.4.12.  They also contain a number of contributed
    packages, found in the subdirectories pkg and small of the main
    distribution directory. Each package is a separate work for copyright
    and license purposes, included in this archive for the convenience of
    the user. The main GAP system is distributed under the GPL version 2,
    packages may have other licenses. Check the documentation and/or the
    website for more details.
    

    To get the upgraded spkg gap-4.4.12 to build, you might also need to edit the installation script “gap-4.4.12/spkg-install”. The above instructions can be adjusted for any spkg’s, whether that be in the standard, optional or experimental repository.

    You might also want to start a new spkg, one which is not to be found in any of the standard, optional or experimental repositories. The general process for creating a completely new spkg is documented in the Developers’ Guide, see especially the section “Producing New Sage Packages”. However, I think the process as documented in that section misses a crucial step, i.e. using the command hg to create a completely new Mercurial repository. Say there is a software package called mypackage-x.y.z and you want to produce an spkg out of it. Rename mypackage-x.y.z to src, and create a directory named mypackage-x.y.z. Yes, this is the same directory name as that which you renamed to src earlier. But now mypackage-x.y.z/ is an empty directory, whereas src/ contains the source code of the upstream project. Navigate to mypackage-x.y.z/ and create the files .hgignore, spkg-install and SPKG.txt, following the general structure as documented at http://www.sagemath.org/doc/developer/producing_spkgs.html. Next, use hg to initialize a new Mercurial repository and add the files .hgignore, spkg-install and SPKG.txt to the repository. Then use hg to check in your changes:

    $ cd mypackage-x.y.z/
    $ # create files .hgignore, spkg-install, SPKG.txt
    $ # add content to these files, following documented structure
    $ hg init
    $ hg add .hgignore spkg-install SPKG.txt
    $ hg commit
    

    Mercurial might not be installed on your system. However, if you have a working Sage installation on your system, you could use the version of Mercurial as shipped with Sage. Suppose Sage is installed under

    /home/username/sage-x.y.z
    

    Now create an alias to the sage script for invoking hg:

    alias 'hg'='/home/username/sage-x.y.z/sage -hg'
    

    and you’re done. Then follow the steps for using Mercurial as described above.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: