scitools.Regression

Regression module for automating regression tests.

Class	Description
TestRun	utilities for easy set up of regression tests
TestRunNumerics	extensions of TestRun for numerical computing
Verify	search for test scripts, run them, compare new results with reference results
VerifyDiffpack	tailored Verify for Diffpack (incl. compilation)
FloatDiff	visual diff between files with real numbers

Usage

Let us start with a very simple Python script, circle.py, for which we would like to create a regression test:

#!/usr/bin/env python
'''Numerical integration of circular motion.'''
import math, sys, os

R=1; w=2*math.pi;  # global constants
def advance(x, y, dt, t):
    '''advance (x,y) point one time step dt with Forward Euler,
       the equations describe circular motion of a body:
       dx/dt = -w*R*cos(2pi*w*t), dy/dt = w*R*sin(2pi*w*t)
    '''
    x = x - dt*w*R*math.sin(w*t);  y = y + dt*w*R*math.cos(w*t)
    return (x,y)

# integrate from 0 to tstop
try:
    tstop = float(sys.argv[1]); dt = float(sys.argv[2])
except:
    print 'Usage: %s tstop dt' % sys.argv[0]; sys.exit(1)

# make output format compatible with the plotpairs.py script:
xmax = R*1.8; xmin = -xmax; ymin = xmin; ymax = xmax
print xmin, xmax, ymin, ymax
n = int(tstop/dt) + 1;
x = R; y = 0
for i in range(n):
    t = i*dt
    x, y = advance(x, y, dt, t)
    print x, y
print 'end'

This script takes two input parameters from the command line: the stop time for the simulation, and the time step. The output is a series of numbers. We may run the script as:

unix> python circle.py 3 0.1 > result

Let us assume that we are sure that the produced results in this case are correct. A regression test then means that we can automatically (in the future) run the above command and compare the new results with archived correct results.

The procedure for using the Regression module in this examples goes as follows.

1. Create a name of the test, say “circle”.
2. Create circle.verify for running the test and saving desired results in a file circle.v.
3. If we believe the results are correct, copy circle.v to circle.r. This circle.r file represents the archived correct results.

Later, we may run:

unix> regression.py verify circle

to rerun the test and investigate differences between circle.v (new results) and circle.r (archived correct results). A whole directory tree can be examined for tests (.verify files) by a similar command:

unix> regression.py verify root-of-tree

Sometimes the new results are correct, but there are significant differences between old and new results, because of, e.g., a change in output formatting. New results can in this case be updated to archive status by:

unix> regression.py update root-of-tree

For the circle.py script, a typical circle.verify script takes the following trivial form if we write it in bash:

#!/bin/sh
./circle.py 3 0.1 > circle.v

The Regression module has tools for running programs, automatically measuring CPU time, selecting lines from a file to put in the .v file, ignoring round-off errors in numerical results, etc. A more sophisticated test, which also accounts for numerical precision in the output, goes as follows (to understand the various statements you will need to consult Appendix B.4 in the textbook “Python Scripting for Computational Science”, by H. P. Langtangen):

#!/usr/bin/env python
import os, sys
from Regression import TestRunNumerics, defaultfilter
test = TestRunNumerics('circle2.v')
test.run('circle.py', options='1 0.21')
# truncate numerical expressions in the output:
test.approx(defaultfilter)
# generate circle2.vd file in correct format:
fd = open('circle2.vd', 'w')
fd.write('## exact data

‘)

# grab the output from circle.py, throw away the # first and last line, and merge the numbers into # one column: cmd = ‘circle.py 1 0.21’ output = os.popen(cmd) res = output.readlines() output.close() numbers = [] for line in res[1:-1]: # skip first and last line

for r in line.split():

numbers.append(r)

# dump length of numbers and its contents: fd.write(‘%d

‘ % len(numbers))

for r in numbers: fd.write(r + ‘

‘)

fd.close()

This sample script can be adapted to a wide range of cases.

class scitools.Regression.FloatDiff(master, file1, file2)

Methods

buildGUI(list, file1, file2)	Build list of fields and canvas (text and plot).
highlight(text, s1, s2)
lift(win1, win2)
loadfiles(file1, file2)	Purpose: build “list”, a list of relevant info for
selectfield()

__init__(master, file1, file2)

__module__ = 'scitools.Regression'

buildGUI(list, file1, file2)

Build list of fields and canvas (text and plot).

highlight(text, s1, s2)

lift(win1, win2)

loadfiles(file1, file2)

Purpose: build “list”, a list of relevant info for all differences found between file1 and file2 (list can afterwards be visualized in buildGUI).

selectfield()

class scitools.Regression.TestRun(logfile, removepath=' ')

Utility for writing individual regression tests. Example on usage: from scitools.Regression import TestRun test = TestRun(“test1.v”) # run a program, place output on test1.v: test.run(“myprog”, options=”-g -p 3.2”, inputfile=”test1.i”) # append a file to test1.v: test.append(“res1.dat”)

Methods

append(file[, maxlines])	Append a file or a list of files to the logfile.
graphics(program[, options])	Run e.g.
grepfile(regex, file[, return_lineinfo])	Return a text consisting of the lines matching regex
loadfile(file)	Return a file as a list of lines for text processing.
movie(list_of_psfiles)	Insert HTML commands for an animated gif picture.
picture(psfile)	Insert HTML commands for a gif picture.
run(program[, options, inputfile])	Run program, store output on logfile.
silentrun(program[, options])	Run program without storing output on logfile.
write(text)	Write message to logfile.

__init__(logfile, removepath=' ')

Clean up logfile, create a new one with header.

__module__ = 'scitools.Regression'

append(file, maxlines=0)

Append a file or a list of files to the logfile.

graphics(program, options='')

Run e.g. graphics program if the environment variable BATCH_REGRESSION is not set.

grepfile(regex, file, return_lineinfo=1)

Return a text consisting of the lines matching regex (regex can be string or list of strings, and return_lineinfo is true if each matched line is prefixed with the filename and the line file as a list of lines for text processing).

loadfile(file)

Return a file as a list of lines for text processing.

movie(list_of_psfiles)

Insert HTML commands for an animated gif picture.

picture(psfile)

Insert HTML commands for a gif picture.

run(program, options='', inputfile='')

Run program, store output on logfile.

silentrun(program, options='')

Run program without storing output on logfile.

write(text)

Write message to logfile.

class scitools.Regression.TestRunNumerics(logfile, removepath=' ')

Bases: scitools.Regression.TestRun

Extends class TestRun with an additional logfile for dump of large sets of floating-point numbers for a second-level regression test.

Methods

append(file[, maxlines])	Append a file or a list of files to the logfile.
approx(realfilter)	Run through logfile, find all real numbers and replace
floatdump(program[, options])	Run program and direct output to self.floatlogfile.
graphics(program[, options])	Run e.g.
grepfile(regex, file[, return_lineinfo])	Return a text consisting of the lines matching regex
loadfile(file)	Return a file as a list of lines for text processing.
movie(list_of_psfiles)	Insert HTML commands for an animated gif picture.
picture(psfile)	Insert HTML commands for a gif picture.
run(program[, options, inputfile])	Run program, store output on logfile.
silentrun(program[, options])	Run program without storing output on logfile.
write(text)	Write message to logfile.

__init__(logfile, removepath=' ')

__module__ = 'scitools.Regression'

append(file, maxlines=0)

Append a file or a list of files to the logfile.

approx(realfilter)

Run through logfile, find all real numbers and replace them by approximate real numbers computed by realfilter.

floatdump(program, options='')

Run program and direct output to self.floatlogfile.

graphics(program, options='')

Run e.g. graphics program if the environment variable BATCH_REGRESSION is not set.

grepfile(regex, file, return_lineinfo=1)

Return a text consisting of the lines matching regex (regex can be string or list of strings, and return_lineinfo is true if each matched line is prefixed with the filename and the line file as a list of lines for text processing).

loadfile(file)

Return a file as a list of lines for text processing.

movie(list_of_psfiles)

Insert HTML commands for an animated gif picture.

picture(psfile)

Insert HTML commands for a gif picture.

run(program, options='', inputfile='')

Run program, store output on logfile.

silentrun(program, options='')

Run program without storing output on logfile.

write(text)

Write message to logfile.

class scitools.Regression.Verify(root='.', task='verify', diffsummary='verify_log', diffprog=None)

Automates regression tests by running through a directory tree, searching for .verify files, executing them and comparing .v with .r files. The result of the comparison (the differing lines) are reported in HTML documents.

Methods

clean(dirname)
diffCPUonly(difflines)
run(scriptfile)

__init__(root='.', task='verify', diffsummary='verify_log', diffprog=None)

__module__ = 'scitools.Regression'

clean(dirname)

diffCPUonly(difflines)

run(scriptfile)

class scitools.Regression.VerifyDiffpack(root='.', task='verify', diffsummary='verify_log', diffprog=None, makemode='opt')

Bases: scitools.Regression.Verify

Extend class Verify with compilation of Diffpack applications, clean-up of applications, and a parameter self.makemode for running applications in opt or nopt mode.

Methods

clean(dirname)	Clean up files, typically executables etc.
diffCPUonly(difflines)
run(scriptfile)	Run script, but compile the application first.

__init__(root='.', task='verify', diffsummary='verify_log', diffprog=None, makemode='opt')

__module__ = 'scitools.Regression'

clean(dirname)

Clean up files, typically executables etc.

diffCPUonly(difflines)

run(scriptfile)

Run script, but compile the application first.

scitools.Regression.defaultfilter(r)

scitools.Regression.exactfilter(r)

scitools.Regression.verify_file_template(casename, floats=False)

scitools.Regression.walk(top, func, arg)

Simple copy of os.path.walk, but does not break when a file marked for visit is deleted during the process.