Much of the material in this document is taken from Appendix H.6 in the book A Primer on Scientific Programming with Python, 4th edition, by the same author, published by Springer, 2014.

Table of contents

Requirements of the test function
Writing the test function; precomputed data
Writing the test function; exact numerical solution
Testing of function robustness
Automatic execution of tests

Model software. We need a piece of software we want to test. Here we choose a function that runs Newton's method for solving algebraic equations \( f(x)=0 \). A very simple implementation goes like

def Newton_basic(f, dfdx, x, eps=1E-7):
    n = 0  # iteration counter
    while abs(f(x)) > eps:
        x = x - f(x)/dfdx(x)
        n += 1
    return x, f(x), n

Requirements of the test function

The simplest way of using the pytest or nose testing frameworks is to write a set of test functions, scattered around in files, such that pytest or nose can automatically find and run all the test functions. To this end, the test functions need to follow certain conventions.

There are many ways of raising the AssertionError exception:

# Formulate a test
tol = 1E-14              # comparison tolerance for real numbers
success = abs(reference - result) < tol
msg = 'computed_result=%d != %d' % (result, reference)

# Explicit raise
if not success:
    raise AssertionError(msg)

# assert statement
assert success, msg

# nose tools
import nose.tools as nt
nt.assert_true(success, msg)
# or
nt.assert_almost_equal(result, reference, msg=msg, delta=tol)

This document contains a lot of test functions following the conventions of the pytest and nose testing frameworks, and we almost exclusively use the plain assert statement to have full control of what the test method is. In more complicated software the many functions in nose.tools may save quite some coding and are convenient to use.

Writing the test function; precomputed data

Newton's method for solving an algebraic equation \( f(x)=0 \) results in only an approximate root \( x_r \), making \( f(x_r)\neq 0 \), but \( |f(x_r)|\leq \epsilon \), where \( \epsilon \) is supposed to be a prescribed number close to zero. The problem is that we do not know beforehand what \( x_r \) and \( f(x_r) \) will be. However, if we strongly believe the function we want to test is correctly implemented, we can record the output from the function in a test case and use this output as a reference for later testing.

Assume we try to solve \( \sin(x) = 0 \) with \( x=-\pi/3 \) as start value. Running Newton_basic with a moderate-size eps (\( \epsilon \)) of \( 10^{-2} \) gives \( x=0.000769691024206 \), \( f(x)=0.000769690948209 \), and \( n=3 \). A test function can now compare new computations with these reference results. Since new computations on another computer may lead to round-off errors, we must compare real numbers with a small tolerance:

def test_Newton_basic_precomputed():
    from math import sin, cos, pi

    def f(x):
        return sin(x)

    def dfdx(x):
        return cos(x)

    x_ref = 0.000769691024206
    f_x_ref = 0.000769690948209
    n_ref = 3

    x, f_x, n = Newton_basic(f, dfdx, x=-pi/3, eps=1E-2)

    tol = 1E-15  # tolerance for comparing real numbers
    assert abs(x_ref - x) < tol       # is x correct?
    assert abs(f_x_ref - f_x) < tol   # is f_x correct?
    assert n == 3                     # is n correct?

The assert statements involving comparison of real numbers can alternatively be carried out by nose.tools functionality:

nose.tools.assert_almost_equal(x_ref, x, delta=tol)

For simplicity we dropped the optional messages explaining what wen wrong if tests fail.

Writing the test function; exact numerical solution

Approximate numerical methods are sometimes exact in certain special cases. An exact answer known beforehand is a good starting point for a test since the implementation should reproduce the known answer to machine precision. For Newton's method we know that it finds the exact root of \( f(x)=0 \) in one iteration if \( f(x) \) is a linear function of \( x \). This fact leads us to a test with \( f(x)=ax+b \), where we can choose \( a \) and \( b \) freely, but it is always wise to choose numbers different from 0 and 1 since these have special arithmetic properties that can hide programming errors.

The test function contains the problem setup, a call to the function to be verified, and assert tests on the output, this time also with an error message in case tests fail:

def test_Newton_basic_linear():
    """Test that a linear function is handled in one iteration."""
    f = lambda x: a*x + b
    dfdx = lambda x: a
    a = 0.25; b = -4
    x_exact = 16
    eps = 1E-5
    x, f_x, n = Newton_basic(f, dfdx, -100, eps)

    tol = 1E-15  # tolerance for comparing real numbers
    assert abs(x - 16) < tol, 'wrong root x=%g != 16' % x
    assert abs(f_x) < eps, '|f(root)|=%g > %g' % (f_x, eps)
    assert n == 1, 'n=%d, but linear f should have n=1' % n

Testing of function robustness

Our Newton_basic function is very basic and suffers from several problems:

• for divergent iterations it will iterate forever,
• it can divide by zero in f(x)/dfdx(x),
• it can perform integer division in f(x)/dfdx(x),
• it does not test whether the arguments have acceptable types and values.

def Newton(f, dfdx, x, eps=1E-7, maxit=100):
    if not callable(f):
        raise TypeError(
            'f is %s, should be function or class with __call__'
            % type(f))
    if not callable(dfdx):
        raise TypeError(
            'dfdx is %s, should be function or class with __call__'
            % type(dfdx))
    if not isinstance(maxit, int):
        raise TypeError('maxit is %s, must be int' % type(maxit))
    if maxit <= 0:
        raise ValueError('maxit=%d <= 0, must be > 0' % maxit)

    n = 0  # iteration counter
    while abs(f(x)) > eps and n < maxit:
        try:
            x = x - f(x)/float(dfdx(x))
        except ZeroDivisionError:
            raise ZeroDivisionError(
            'dfdx(%g)=%g - cannot divide by zero' % (x, dfdx(x)))
        n += 1
    return x, f(x), n

The numerical functionality can be tested as described in the previous example, but we should include additional tests for testing the additional functionality. One can have different tests in different test functions, or collect several tests in one test function. The preferred strategy depends on the problem. Here it may be natural to have different test functions only when the \( f(x) \) formula differs to avoid repeating code.

To test for divergence, we can choose \( f(x) = \tanh(x) \), which is known to lead to divergent iterations if not \( x \) is sufficiently close to the root \( x=0 \). A start value \( x=20 \) reveals that the iterations are divergent, so we set maxit=12 and test that the actual number of iterations reaches this limit. We can also add a test on \( x \), e.g., that \( x \) is a big as we know it will be: \( x>10^{50} \) after 12 iterations. The test function becomes

def test_Newton_divergence():
    from math import tanh
    f = tanh
    dfdx = lambda x: 10./(1 + x**2)

    x, f_x, n = Newton(f, dfdx, 20, eps=1E-4, maxit=12)
    assert n == 12
    assert x > 1E+50

To test for division by zero, we can find an \( f(x) \) and an \( x \) such that \( f'(x)=0 \). One simple example is \( x=0 \), \( f(x)=\cos(x) \), and \( f'(x)=-\sin(x) \). If \( x=0 \) is the start value, we know that a division by zero will take place in the first iteration, and this will lead to a ZeroDivisionError exception. We can explicitly handle this exception and introduce a boolean variable success that is True if the exception is raised and otherwise False. The corresponding test function reads

def test_Newton_div_by_zero1():
    from math import sin, cos
    f = cos
    dfdx = lambda x: -sin(x)
    success = False
    try:
        x, f_x, n = Newton(f, dfdx, 0, eps=1E-4, maxit=1)
    except ZeroDivisionError:
        success = True
    assert success

There is a special nose.tools.assert_raises helper function that can be used to test if a function raises a certain exception. The arguments to assert_raises are the exception type, the name of the function to be called, and all positional and keyword arguments in the function call:

import nose.tools as nt

def test_Newton_div_by_zero2():
    from math import sin, cos
    f = cos
    dfdx = lambda x: -sin(x)
    nt.assert_raises(
        ZeroDivisionError, Newton, f, dfdx, 0, eps=1E-4, maxit=1)

Let us proceed with testing that wrong input is caught by function Newton. Since the same type of exception is raised for different type of errors we shall now also examine (parts of) the exception messages. The first test involves an argument f that is not a function:

def test_Newton_f_is_not_callable():
    success = False
    try:
        Newton(4.2, 'string', 1.2, eps=1E-7, maxit=100)
    except TypeError as e:
        if "f is <type 'float'>" in e.message:
            success = True

As seen, success = True demands that the right exception is raised and that its message starts with f is <type 'float'>. What text to expect in the message is evident from the source in function Newton.

The nose.tools module also has a function for testing the exception type and the message content. This is illustrated when dfdx is not callable:

def test_Newton_dfdx_is_not_callable():
    nt.assert_raises_regexp(
        TypeError, "dfdx is <type 'str'>",
        Newton, lambda x: x**2, 'string', 1.2, eps=1E-7, maxit=100)

Checking that Newton catches maxit of wrong type or with a negative value can be carried out by these test functions:

def test_Newton_maxit_is_not_int():
    nt.assert_raises_regexp(
        TypeError, "maxit is <type 'float'>",
        Newton, lambda x: x**2, lambda x: 2*x,
        1.2, eps=1E-7, maxit=1.2)

def test_Newton_maxit_is_neg():
    nt.assert_raises_regexp(
        ValueError, "maxit=-2 <= 0",
        Newton, lambda x: x**2, lambda x: 2*x,
        1.2, eps=1E-7, maxit=-2)

The corresponding support for testing exceptions in pytest is

import pytest
with pytest.raises(TypeError) as e:
    Newton(lambda x: x**2, lambda x: 2*x, 1.2, eps=1E-7, maxit=-2)

Automatic execution of tests

Our code for the Newton_basic and Newton functions is placed in a file eq_solver.py together with the tests. To run all test functions with names of the form test_*() in this file, use the nosetests or py.test commands, e.g.,:

Terminal> nosetests -s eq_solver.py
..........
-------------------------------------------------------------------
Ran 10 tests in 0.004s

OK

The -s option causes all output from the called functions in the program eq_solver.py to appear on the screen (by default, nosetests and py.test suppress all output). The final OK points to the fact that no test failed. Adding the option -v prints out the outcome of each individual test function. In case of failure, the AssertionError exception and the associated message, if existing, are displayed. Pytest also displays the code that failed.

One can also collect test functions in separate files with names starting with test. A simple command nosetests -s -v will look for all such files in this folder as well as in all subfolders if the folder names start with test or end with _test or _tests. By following this naming convention, nosetests can automatically run a potentially large number of tests and give us quick feedback. The py.test -s -v command will look for and run all test files in the entire tree of any subfolder.

• ← Prev