Archive

Archive for September, 2009

Software Freedom Day 2009

20 September 2009 Leave a comment

The 2009 Software Freedom Day (SFD) took place on Saturday 19th September. The Linux Users of Victoria group organized the celebration at Chadstone Shopping Centre, Melbourne, Australia. I have uploaded some photos taken during the day to picasa. Brianna Laugher has uploaded more photos. Here’s a news item at iTWire about the SFD celebration in Melbourne. I’m delighted to say that it mentions Sage a lot :-)

This year’s venue was more spacious than the venue for the 2008 celebration. Ten speakers delivered a total of 11 half-hour talks in two parallel talk sessions, spread across two lecture rooms. Three workshop demonstrators delivered hands-on tutorials in a computer lab. Here is a table listing the talks and practical workshops.



I gave a talk on beginning programming with Python. I have uploaded the slides for the talk as well as written up notes for it.

Beginning programming with Python

20 September 2009 4 comments

These are notes of a talk I delivered during Software Freedom Day 2009 at Chadstone Shopping Centre, Melbourne, Australia. Slides are also available. This talk requires no previous experiences with programming. After covering some basic principles of programming, these principles are then demonstrated using the Python programming language.

What’s a program?

When you start programming, you are asserting more control over a computer than if you were using the computer. To this end, you need to have a deeper understanding of how computers work.

As a computer user, you use programs that have been written by other people. Such programs include:

  • web browsers, e.g. Firefox, IE, Opera, Safari
  • word processors, e.g. Abiword, OpenOffice, Word
  • multimedia players, e.g. MPlayer, VLC
  • document viewers, e.g. Acrobat Reader, evince, Foxit, okular
  • email clients, e.g. Evolution, Outlook, Thunderbird
  • webmail, e.g. Gmail, Hushmail, Windows Live, Yahoo!

You might use some of these programs throughout a working day, without having to be concerned about the their internals. Why would you want to worry about how a program work when you simply want to use it?

I have referred to various programs, but have not talked about what a program is. Think of a program as a sequence of instructions that specify how to perform a task. The instructions might be about directing a computer to carry out a mathematical calculation, position a window at a certain place on the screen, process some data, or send an email. Regardless of what the purpose of the program is, the instructions that constitute the program must be written in a format comprehensible by a computer.

Programming languages

Regardless of what the program is meant to achieve, you must write it using a programming language. At the lowest level, computers read your program as a bunch of ones and zeros, which is essentially what computers do: they process ones and zeros, or a sequence of bits. For example, “A” might be represented as 01000001, “B” as 01000010, and “C” as 01000011, so “ABC” becomes

\text{ABC} \longrightarrow 010000010100001001000011

But then programming using ones and zeros is next to impossible and very error-prone. To make it easier to program a computer, you can use a high-level programming language. A high-level programming language can look a lot like a natural language such as English to make it easier for people to read and understand what a program does. Examples of high-level programming languages include

  • C, C++
  • Java, JavaScript
  • Python, Ruby
  • Perl, PHP

Programming languages may differ in many ways, but they usually share a number of basic principles:

  • input: Retrieve data from somewhere, e.g. the keyboard, a file, somewhere on the web, a device.
  • output: Display data on the screen, send data to a file, over the web, or to a device.
  • math: Perform basic mathematical operations such as addition, multiplication, subtraction, and division.
  • conditional execution: Check for a condition. If it is true, then execute the sequence of instructions for that condition. If it is false, execute a different sequence of instructions.
  • repetition: Perform a sequence of instructions repeatedly.

These basic principles of programming languages are what constitute a computer program. When writing a program, you mix and match them in certain ways in order that the program carry out the instructions you want it to execute.

Hello, World!

I will be using Python for all code examples from now on. Python is a programming language suitable for teaching and learning how to program. It is also used for writing very complex programs. You can get Python from http://www.python.org.

Upon loading Python from the command line, you would see something like this:

Python 2.6.2 (r262:71600, Jul 10 2009, 10:49:45)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

Now Python is waiting for you to enter some instructions for it to run. The three greater than signs >>> is the Python prompt, awaiting your instructions.

When learning a programming language, one of the first things that people learn is how to display the phrase, “Hello, World!”. This can be easily achieved using Python as follows:

>>> print "Hello, World!"
Hello, World!

Here, print is a statement that tells Python to display “Hello, World!”. It does so, removing the opening and closing quotation marks. The opening and closing quotation marks are used to denote the beginning and end of the text you want to display on the screen, which is why you don’t see these quotation marks in the result. You can also use Python as a calculator to do basic mathematics such as addition, multiplication, division, and subtraction.

>>> 2 + 3   # addition
5
>>> 2 * 3   # multiplication
6
>>> 12 / 4  # division
3
>>> 10 - 5  # subtraction
5

Notice that Python ignores everything after the hash symbol #. The text “# addition” is a comment. It is there to explain what the code statement “2 + 3” is doing. You don’t need to comment every line of your code. Keep in mind that months from now when you again read your code you might have forgotten what it is doing. In programming, precision is as important as thorough documentation.

Put on your problem solving hat

Programming is a creative activity mixed with logical precision. You write a set of precise instructions for a computer to carry out. Many people enjoy programming because it is a way for them to be creative. When things don’t go as intended, you need to put on your detective hat and figure out what went wrong. This process is called debugging.

Syntax error

In the text “Hello, World!”, suppose you leave out the closing quotation mark. This is what would happen if you do so:

>>> print "Hello, World!
  File "", line 1
    print "Hello, World!
                       ^
SyntaxError: EOL while scanning string literal

Python is saying that

print "Hello, World!

is not a valid code statement in the Python programming language. With the text “SyntaxError“, Python is also pointing out the type of error in the above code. In this case, it is a syntax error, meaning that the code violates the syntax of the Python language. In US English, acronyms and abbreviations are spelt with full stops, e.g. U.S.A., Ms. and Mr. In Australian English, you would spell these as USA, Ms and Mr. Even if you spell them as U.S.A., Ms. and Mr. in an email to another Australian person, the person would likely understand what you mean without complaining. But Python is not so forgiving. It would complain whenever your code doesn’t follow the syntax of the Python language.

Runtime error

Another type of error that can lurk in your code is runtime error. Runtime errors are also called exceptions because they indicate that something exceptional has happened. Consider this Python code:

>>> print n
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'n' is not defined

Python is telling you that it doesn’t know what n is. Is n a number? Is it meant to denote the text “Hello, World!“? If n is meant to be holding some value, then it becomes something known as a variable. Think of a variable as a box for storing something so that it can be retrieved later on. You can assign something to n by using the equal symbol =.

>>> n = 23
>>> print n
23

In the above code, the variable n is assigned the number 23. That value is then displayed on the screen.

You can also assign a string to a variable. A string is a sequence of characters. Besides numbers, strings are very useful in many programming languages. You can also assign a string to a variable using the equal symbol:

>>> name = "Wendy Smith"
>>> print name
Wendy Smith

Garbage in, garbage out

Semantic errors usually occur when your program runs successfully, but the program doesn’t do what you intended it to do. This situation is often described as, “Garbage in, garbage out.” Semantic errors can be thought of as the meaning of a sequence of instructions. Do the instructions give you the result they are meant to give?

Here is an example of a semantic error:

>>> name = "123"
>>> print name
123

In this case, the code runs OK. Python doesn’t complain about anything; it has carried out the instructions as you have given. However, the result is not what was intended. The variable name should have referred to a sensible name such as “Wendy Smith“. But instead the variable refers to a number. In a large program, semantic errors can be difficult to find.

Functions

In programming, a function is a way to name a sequence of instructions. With a function, you can run that sequence of instructions over and over again by using the function name, instead of having to type the sequence of instructions every time you want to use the instructions. Suppose you want to write a function that displays the name and age of a person. Let’s give that function the name “person“:

>>> def person(name, age):
...     print "Name is", name
...     print "Age is", age
...
>>> person("Wendy Smith", 23)
Name is Wendy Smith
Age is 23

The word def tells Python that you are starting to define a function. The function name is person, whose parameters are contained within a pair of opening and closing parentheses. The parameters of a function are variables that hold values you pass to the function so that the function can use those values. The function person takes two parameters, i.e. name and age which are both variables. You can then use the function by specifying the function name, followed by the values for the parameters. In this case, the parameter name is given the string “Wendy Smith” and the parameter age is given the number 23.

Functions without parameters

A function doesn’t need to have any parameters. You can define a function that takes no parameters whatsoever and still do something useful.

>>> def recite_poem():
...     print "It's a world of laughter and a world of tears"
...     print "It's a world of hope and a world of fears"
...
>>> recite_poem()
It's a world of laughter and a world of tears
It's a world of hope and a world of fears

And you can recite the poem again and again. Let’s say you want to recite the poem five times. You can write another function that uses the function recite_poem() five times:

>>> def recite_poem():
...     print "It's a world of laughter and a world of tears"
...     print "It's a world of hope and a world of fears"
...
>>> def rehearse():
...     recite_poem()
...     recite_poem()
...     recite_poem()
...     recite_poem()
...     recite_poem()
...
>>> rehearse()
It's a world of laughter and a world of tears
It's a world of hope and a world of fears
It's a world of laughter and a world of tears
It's a world of hope and a world of fears
It's a world of laughter and a world of tears
It's a world of hope and a world of fears
It's a world of laughter and a world of tears
It's a world of hope and a world of fears
It's a world of laughter and a world of tears
It's a world of hope and a world of fears

The function rehearse() certainly does the job if you always want to rehearse the poem exactly five times. What if you want to rehearse the poem more than five times? There should be an easier way to write the function rehearse().

Repeat after me

When you need to perform a task a certain number of times, it usually indicates that you can use a loop to do the task. A loop is a way to run a sequence of instructions for some fixed number of times. One of the most common loop is the “for” loop. Like many other programming languages, in Python you can use the “for” loop to repeatedly run some instructions over and over again.

To recite the poem five times, you can do as follows:

>>> for i in range(5):
...     recite_poem()
...
It's a world of laughter and a world of tears
It's a world of hope and a world of fears
It's a world of laughter and a world of tears
It's a world of hope and a world of fears
It's a world of laughter and a world of tears
It's a world of hope and a world of fears
It's a world of laughter and a world of tears
It's a world of hope and a world of fears
It's a world of laughter and a world of tears
It's a world of hope and a world of fears

Now you can use the “for” loop to rewrite the function rehearse() so that it rehearses the poem for the number of times that you want it to.

>>> def rehearse(n):
...     for i in range(n):
...         recite_poem()
...
>>> rehearse(3)
It's a world of laughter and a world of tears
It's a world of hope and a world of fears
It's a world of laughter and a world of tears
It's a world of hope and a world of fears
It's a world of laughter and a world of tears
It's a world of hope and a world of fears

As you can see, the function rehearse() now has one parameter called n. Given any positive whole number, the function rehearse() would recite the poem that many times. Using a for loop, the function rehearse() is now much shorter and more compact than previously. It is also more general and works for a wider range of situations than before.

If this then that

There are times when you want to run a sequence of instructions only if some condition is true. Control flow allows you to separate your code into different sequences, each of which would be run if a specified condition holds true. A common type of control flow is the if statement. Here’s a simple if statement in Python:

>>> if 3 > 0:
...     print "3 is positive"
...
3 is positive

The condition here is to check that three is greater than zero. This is indeed true, so the message “3 is positive” is displayed.

If this, else that

If a condition is false, you can use the else statement to run another sequence of instructions.

>>> if -5 > 0:
...     print "-5 is positive"
... else:
...     print "-5 is negative"
...
-5 is negative

In this case, -5 is less than zero so the message “-5 is negative” is displayed.

Let’s write a function called check_number() that checks whether a number is positive, negative, or zero.

>>> def check_number(n):
...     if n > 0:
...         print n, "is positive"
...     if n < 0:
...         print n, "is negative"
...     if n == 0:
...         print n, "is zero"
...
>>> check_number(3)
3 is positive
>>> check_number(-5)
-5 is negative
>>> check_number(0)
0 is zero

A whole number is either positive, negative or zero. The code takes care of each of those three cases. Notice that the double-equal sign == means “is equal to”, whereas the single-equal sign = means “assign the value on the right to the variable on the left”. So if you want to test whether or not two things are equal to each other, use the double-equal sign ==.

If, else if, else

The function check_number() does what you want. But you can write it in a different way using all of the three control flow statements if, else and elif. The statement elif is short for “else if”. Using those three control flow statements, you can rewrite the function check_number() as follows:

>>> def check_number(n):
...     if n > 0:
...         print n, "is positive"
...     elif n < 0:
...         print n, "is negative"
...     else:
...         print n, "is zero"
...
>>> check_number(13)
13 is positive
>>> check_number(-7)
-7 is negative
>>> check_number(0)
0 is zero

The rewritten function works as expected and doesn’t use lots of if statements.

What’s after this?

I have talked about some basic principles that are common to many programming languages. These include input, output, mathematics, control flow, and repetition. Each of these five principles were demonstrated using the Python programming language. Here I list some resources that talk about beginning programming with Python.

  1. Allen B. Downey. Think Python: How to Think Like a Computer Scientist.
  2. Peter Norton, Alex Samuel, David Aitel, Eric Foster-Johnson, Leonard Richardson, Jason Diamond, Aleatha Parker, Michael Roberts. Beginning Python. Wiley Publishing, 2005.
  3. Mark Pilgrim. Dive Into Python. Apress, 2004.
  4. The Python Tutorial. Python Software Foundation, 2009.

Parallel testing the Sage library

5 September 2009 Leave a comment

After compiling a source version of Sage, you can choose to run the test suite on the whole Sage library, on all modules under a given directory, or on a specified module only. For the purposes of this post, I have compiled Sage 4.1.1 from source and the top level Sage directory is

[mvngu@mod sage-4.1.1]$ pwd
/scratch/mvngu/build/sage-4.1.1

Testing a module

Say you want to run all tests in the sudoku module sage/games/sudoku.py. In a terminal window, first cd to the top level Sage directory of your local Sage installation. Now start doctesting as demonstrated in the following terminal session:

[mvngu@mod sage-4.1.1]$ ./sage -t devel/sage-main/sage/games/sudoku.py
sage -t  "devel/sage-main/sage/games/sudoku.py"             
	 [6.0 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 6.0 seconds

As you can see, the numbers output by the test tell you that testing the sudoku module takes about six seconds, while testing all the modules you specified took the same amount of time. In this case, you only tested one module so it’s not surprising that the total testing time is approximately the same as the time required to test only that one module. Notice that the syntax is

[mvngu@mod sage-4.1.1]$ ./sage -t devel/sage-main/sage/games/sudoku.py
sage -t  "devel/sage-main/sage/games/sudoku.py"             
	 [5.7 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 5.7 seconds
[mvngu@mod sage-4.1.1]$ ./sage -t "devel/sage-main/sage/games/sudoku.py"
sage -t  "devel/sage-main/sage/games/sudoku.py"             
	 [5.4 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 5.4 seconds

and not

[mvngu@mod sage-4.1.1]$ ./sage -t sage/games/sudoku.py
ERROR: File ./sage/games/sudoku.py is missing
exit code: 1
 
----------------------------------------------------------------------
The following tests failed:


	./sage/games/sudoku.py
Total time for all tests: 0.0 seconds
[mvngu@mod sage-4.1.1]$ ./sage -t "sage/games/sudoku.py"
ERROR: File ./sage/games/sudoku.py is missing
exit code: 1
 
----------------------------------------------------------------------
The following tests failed:


	./sage/games/sudoku.py
Total time for all tests: 0.0 seconds

In all of the above terminal sessions, you use a local installation of Sage to test its own modules. Even if you have a system-wide Sage installation, using that version to doctest the modules of a local installation is a recipe for confusion. For example, in the following session, there is a system-wide Sage. Doctesting using that version doesn’t output anything useful.

[mvngu@mod sage-4.1.1]$ which sage
/usr/local/bin/sage
[mvngu@mod sage-4.1.1]$ sage devel/sage-main/sage/games/sudoku.py

Troubleshooting

If you want to doctest modules of a Sage installation, it is recommended that you first cd to the top level directory of that Sage installation, otherwise known as the SAGE_ROOT of that installation. When you run tests, use that particular Sage installation via the syntax ./sage; notice the “dot-forward-slash” at the front of sage. This is a precaution against confusion that can arise when your system has multiple Sage installations. For example, the following syntax is OK because you explicitly specify the Sage installation in the current SAGE_ROOT:

[mvngu@sage sage-4.1.1]$ ./sage -t devel/sage-main/sage/games/sudoku.py 
sage -t  "devel/sage-main/sage/games/sudoku.py"             
	 [5.1 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 5.1 seconds
[mvngu@sage sage-4.1.1]$ ./sage -t "devel/sage-main/sage/games/sudoku.py"
sage -t  "devel/sage-main/sage/games/sudoku.py"             
	 [5.0 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 5.0 seconds

If you login as a regular user, the following syntax is not recommended as you are using a system-wide Sage installation (if it exists):

[mvngu@sage sage-4.1.1]$ sage -t devel/sage-main/sage/games/sudoku.py
Traceback (most recent call last):
  File "/usr/local/sage/local/bin/sage-test", line 49, in 
    os.makedirs(TMP)
  File "/usr/local/sage/local/lib/python/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/sage/tmp/tmp'
[mvngu@sage sage-4.1.1]$ sage -t "devel/sage-main/sage/games/sudoku.py"
Traceback (most recent call last):
  File "/usr/local/sage/local/bin/sage-test", line 49, in 
    os.makedirs(TMP)
  File "/usr/local/sage/local/lib/python/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/sage/tmp/tmp'

In this case, you received a permission error because the system-wide Sage installation attempts to write some data to a system-wide directory using your login privileges. The system-wide directory is a temporary directory under the system-wide SAGE_ROOT. Most likely a system-wide Sage installation was performed by a system administrator using an account with more privileges than a regular user. As a regular user, you cannot write to directories where you don’t have write permission.

The following syntax is also discouraged when you login as a regular user:

[mvngu@sage sage-4.1.1]$ cd
[mvngu@sage ~]$ sage -t devel/sage-main/sage/games/sudoku.py
Traceback (most recent call last):
  File "/usr/local/sage/local/bin/sage-test", line 49, in 
    os.makedirs(TMP)
  File "/usr/local/sage/local/lib/python/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/sage/tmp/tmp'
[mvngu@sage ~]$ sage -t "devel/sage-main/sage/games/sudoku.py"
Traceback (most recent call last):
  File "/usr/local/sage/local/bin/sage-test", line 49, in 
    os.makedirs(TMP)
  File "/usr/local/sage/local/lib/python/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/sage/tmp/tmp'

This is exactly the same as the previous syntax because in both cases you attempted to doctest modules in a system-wide Sage installation using privileges of a regular user. If you don’t have permission to read or write somewhere on your system, you can’t read or write there. As a regular user, you don’t usually have privileges to read the directory /root nor do you have privileges to write to the root directory:

[mvngu@sage sage-4.1.1]$ ls /root/
ls: cannot open directory /root/: Permission denied
[mvngu@sage sage-4.1.1]$ cd /
[mvngu@sage /]$ touch demo.txt
touch: cannot touch `demo.txt': Permission denied

Parallel testing many modules

So far you have used a single thread to doctest a module in the Sage library. There are hundreds, even thousands of modules in the Sage library. Testing them all using one thread would take a few hours. Depending on your hardware, this could take up to six hours or more. On a multi-core system, parallel doctesting can significantly reduce the testing time. Unless you also want to use your computer while doctesting in parallel, you can choose to devote all the cores of your system for parallel testing.

Let’s doctest all modules in a directory, first using a single thread and then using two threads. For this example, suppose you want to test all the modules under sage/crypto/. You can use a syntax similar to that shown above to achieve this:

[mvngu@mod sage-4.1.1]$ ./sage -t devel/sage-main/sage/crypto/
sage -t  "devel/sage-main/sage/crypto/lfsr.py"              
	 [2.5 s]
sage -t  "devel/sage-main/sage/crypto/cryptosystem.py"      
	 [1.9 s]
sage -t  "devel/sage-main/sage/crypto/block_cipher/miniaes.py"
	 [2.5 s]
sage -t  "devel/sage-main/sage/crypto/block_cipher/all.py"  
	 [0.1 s]
sage -t  "devel/sage-main/sage/crypto/block_cipher/__init__.py"
	 [0.1 s]
sage -t  "devel/sage-main/sage/crypto/classical.py"         
	 [2.7 s]
sage -t  "devel/sage-main/sage/crypto/mq/mpolynomialsystem.py"
	 [8.7 s]
sage -t  "devel/sage-main/sage/crypto/mq/mpolynomialsystemgenerator.py"
	 [1.9 s]
sage -t  "devel/sage-main/sage/crypto/mq/__init__.py"       
	 [0.1 s]
sage -t  "devel/sage-main/sage/crypto/mq/sbox.py"           
	 [2.8 s]
sage -t  "devel/sage-main/sage/crypto/mq/sr.py"             
	 [4.9 s]
sage -t  "devel/sage-main/sage/crypto/stream_cipher.py"     
	 [1.9 s]
sage -t  "devel/sage-main/sage/crypto/all.py"               
	 [0.1 s]
sage -t  "devel/sage-main/sage/crypto/stream.py"            
	 [1.9 s]
sage -t  "devel/sage-main/sage/crypto/__init__.py"          
	 [0.1 s]
sage -t  "devel/sage-main/sage/crypto/classical_cipher.py"  
	 [1.9 s]
sage -t  "devel/sage-main/sage/crypto/cipher.py"            
	 [1.9 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 35.7 seconds

Now let’s do the same thing, but this time let’s also use the optional argument -long:

[mvngu@mod sage-4.1.1]$ ./sage -t -long devel/sage-main/sage/crypto/
sage -t -long "devel/sage-main/sage/crypto/lfsr.py"         
	 [1.9 s]
sage -t -long "devel/sage-main/sage/crypto/cryptosystem.py" 
	 [2.0 s]
sage -t -long "devel/sage-main/sage/crypto/block_cipher/miniaes.py"
	 [2.6 s]
sage -t -long "devel/sage-main/sage/crypto/block_cipher/all.py"
	 [0.1 s]
sage -t -long "devel/sage-main/sage/crypto/block_cipher/__init__.py"
	 [0.1 s]
sage -t -long "devel/sage-main/sage/crypto/classical.py"    
	 [2.7 s]
sage -t -long "devel/sage-main/sage/crypto/mq/mpolynomialsystem.py"
	 [8.7 s]
sage -t -long "devel/sage-main/sage/crypto/mq/mpolynomialsystemgenerator.py"
	 [2.2 s]
sage -t -long "devel/sage-main/sage/crypto/mq/__init__.py"  
	 [0.1 s]
sage -t -long "devel/sage-main/sage/crypto/mq/sbox.py"      
	 [2.9 s]
sage -t -long "devel/sage-main/sage/crypto/mq/sr.py"        
	 [56.6 s]
sage -t -long "devel/sage-main/sage/crypto/stream_cipher.py"
	 [2.5 s]
sage -t -long "devel/sage-main/sage/crypto/all.py"          
	 [0.1 s]
sage -t -long "devel/sage-main/sage/crypto/stream.py"       
	 [1.9 s]
sage -t -long "devel/sage-main/sage/crypto/__init__.py"     
	 [0.1 s]
sage -t -long "devel/sage-main/sage/crypto/classical_cipher.py"
	 [1.9 s]
sage -t -long "devel/sage-main/sage/crypto/cipher.py"       
	 [1.9 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 88.0 seconds

Notice the time difference between the first set of tests and the second set, which uses the optional argument -long. Many tests in the Sage library are flagged with “# long time” because these are known to take a long time to run through. Without using the optional -long argument, the module sage/crypto/mq/sr.py took about five seconds. With this optional argument, it required 57 seconds to run through all tests in that module. Here is a snippet of a function in the module sage/crypto/mq/sr.py with a doctest that has been flagged as taking a long time:

def test_consistency(max_n=2, **kwargs):
    r"""
    Test all combinations of ``r``, ``c``, ``e`` and ``n`` in ``(1,
    2)`` for consistency of random encryptions and their polynomial
    systems. `\GF{2}` and `\GF{2^e}` systems are tested. This test takes
    a while.

    INPUT:

    - ``max_n`` - maximal number of rounds to consider (default: 2)
    - ``kwargs`` - are passed to the SR constructor

    TESTS::

        sage: from sage.crypto.mq.sr import test_consistency
        sage: test_consistency(1) # long time -- calling w/ max_n = 2 requires a LOT of RAM (&gt;&gt; 2GB, evidently).  Calling w/ max_n = 1 is far more manageable.
        True

    The above doctest used to fail on a machine with "only" 2GB RAM.
    Using ``max_n = 1`` appears to be a more reasonable memory usage.
    """

Now doctest the same directory in parallel using two threads:

[mvngu@mod sage-4.1.1]$ ./sage -tp 2 devel/sage-main/sage/crypto/
Global iterations: 1
File iterations: 1
Using cached timings to run longest doctests first.
Doctesting 17 files doing 2 jobs in parallel
sage -t  devel/sage-main/sage/crypto/lfsr.py
	 [2.7 s]
sage -t  devel/sage-main/sage/crypto/cryptosystem.py
	 [2.0 s]
sage -t  devel/sage-main/sage/crypto/mq/mpolynomialsystem.py
	 [9.4 s]
sage -t  devel/sage-main/sage/crypto/mq/sr.py
	 [5.2 s]
sage -t  devel/sage-main/sage/crypto/classical.py
	 [2.8 s]
sage -t  devel/sage-main/sage/crypto/mq/sbox.py
	 [3.2 s]
sage -t  devel/sage-main/sage/crypto/block_cipher/miniaes.py
	 [2.6 s]
sage -t  devel/sage-main/sage/crypto/stream_cipher.py
	 [2.0 s]
sage -t  devel/sage-main/sage/crypto/mq/mpolynomialsystemgenerator.py
	 [2.0 s]
sage -t  devel/sage-main/sage/crypto/classical_cipher.py
	 [2.1 s]
sage -t  devel/sage-main/sage/crypto/cipher.py
	 [2.1 s]
sage -t  devel/sage-main/sage/crypto/__init__.py
	 [0.1 s]
sage -t  devel/sage-main/sage/crypto/block_cipher/__init__.py
	 [0.1 s]
sage -t  devel/sage-main/sage/crypto/mq/__init__.py
	 [0.1 s]
sage -t  devel/sage-main/sage/crypto/block_cipher/all.py
	 [0.1 s]
sage -t  devel/sage-main/sage/crypto/stream.py
	 [2.0 s]
sage -t  devel/sage-main/sage/crypto/all.py
	 [0.1 s]
 
----------------------------------------------------------------------
All tests passed!
Timings have been updated.
Total time for all tests: 19.3 seconds

[mvngu@mod sage-4.1.1]$ ./sage -tp 2 -long devel/sage-main/sage/crypto/
Global iterations: 1
File iterations: 1
No long cached timings exist; will create upon successful finish.
Doctesting 17 files doing 2 jobs in parallel
sage -t -long devel/sage-main/sage/crypto/cryptosystem.py
	 [2.7 s]
sage -t -long devel/sage-main/sage/crypto/lfsr.py
	 [2.7 s]
sage -t -long devel/sage-main/sage/crypto/stream_cipher.py
	 [2.2 s]
sage -t -long devel/sage-main/sage/crypto/all.py
	 [0.1 s]
sage -t -long devel/sage-main/sage/crypto/classical.py
	 [3.0 s]
sage -t -long devel/sage-main/sage/crypto/__init__.py
	 [0.1 s]
sage -t -long devel/sage-main/sage/crypto/stream.py
	 [2.1 s]
sage -t -long devel/sage-main/sage/crypto/classical_cipher.py
	 [2.1 s]
sage -t -long devel/sage-main/sage/crypto/cipher.py
	 [2.1 s]
sage -t -long devel/sage-main/sage/crypto/block_cipher/all.py
	 [0.1 s]
sage -t -long devel/sage-main/sage/crypto/block_cipher/__init__.py
	 [0.1 s]
sage -t -long devel/sage-main/sage/crypto/block_cipher/miniaes.py
	 [2.8 s]
sage -t -long devel/sage-main/sage/crypto/mq/mpolynomialsystemgenerator.py
	 [2.0 s]
sage -t -long devel/sage-main/sage/crypto/mq/__init__.py
	 [0.1 s]
sage -t -long devel/sage-main/sage/crypto/mq/sbox.py
	 [3.1 s]
sage -t -long devel/sage-main/sage/crypto/mq/mpolynomialsystem.py
	 [9.1 s]
sage -t -long devel/sage-main/sage/crypto/mq/sr.py
	 [56.0 s]
 
----------------------------------------------------------------------
All tests passed!
Timings have been updated.
Total time for all tests: 71.8 seconds

As the number of threads increases, the total testing time decreases. To minimize confusion, it’s also a good idea to explicitly specify the path name of the directory you want to doctest and not a symbolic link to that directory. In the above examples, the symbolic link devel/sage points to the directory devel/sage-main, but the actual path to the directory has been specified instead of its symbolic link.

Parallel testing the whole Sage library

The main Sage library resides in the directory devel/sage-main/. You can use the syntax described above to doctest the main library using multiple threads. When doing release management or patching the main Sage library, I would parallel test the library using ten threads:

[mvngu@mod sage-4.1.1]$ ./sage -tp 10 -long devel/sage-main/

Another way is to edit the file makefile in the top level Sage directory so that the variable NUM_THREADS is set to 10:

# How many threads should be used when doing parallel testing (and              
# sometime in the future, parallel building)?                                   
NUM_THREADS=10

After saving all changes to makefile, you can parallel test with the -long option using ten threads:

[mvngu@mod sage-4.1.1]$ make ptestlong

Any of the following commands would also doctest the Sage library or one of its clones:

make test
make check
make testlong
make ptest
make ptestlong

In each case, testing is performed on the directory that is pointed to by the symbolic link devel/sage.




Updates 2010-01-08 — This post is now in the Sage Developers’ Guide.

Follow

Get every new post delivered to your Inbox.