This chapter is taken from book A Primer on Scientific Programming with Python by H. P. Langtangen, 4th edition, Springer, 2014.
The simplest way of running another program from Python is to
use os.system:
cmd = 'python myprog.py 21 --mass 4' # command to be run
failure = os.system(cmd)
if failure:
print 'Execution of "%s" failed!\n' % cmd
sys.exit(1)
The recommended way to run operating system commands is to use
the subprocess module. The above command is equivalent to
import subprocess
cmd = 'python myprog.py 21 --mass 4'
failure = subprocess.call(cmd, shell=True)
# or
failure = subprocess.call(
['python', 'myprog.py', '21', '--mass', '4'])
The output of an operating system command can be stored in a string object:
try:
output = subprocess.check_output(cmd, shell=True,
stderr=subprocess.STDOUT)
except subprocess.CalledProcessError:
print 'Execution of "%s" failed!\n' % cmd
sys.exit(1)
# Process output
for line in output.splitlines():
...
The stderr argument ensures that the output
string contains everything that the command cmd wrote
to both standard output and standard error.
The constructions above are mainly used for running stand-alone programs.
Any file or folder listing or manipulation should be done by the
functionality in the os and shutil modules.
Given data/file1.dat as a file path relative to the
home folder /users/me ($HOME/data/file1.dat in Unix).
Python has tools for extracting the complete folder name
/users/me/data, the basename file1.dat, and the extension
.dat:
>>> path = os.path.join(os.environ['HOME'], 'data', 'file1.dat')
>>> path
'/users/me/data/file1.dat'
>>> foldername, basename = os.path.split(path)
>>> foldername
'/users/me/data'
>>> basename
'file1.dat'
>>> stem, ext = os.path.splitext(basename)
>>> stem
'file1'
>>> ext
'.dat'
>>> outfile = stem + '.out'
>>> outfile
'file1.out'
Arguments to Python functions are of four types:
def f(pos1, pos2, key1=val1, key2=val2, *args, **kwargs):
Here, pos1 and pos2 are positional arguments,
key1 and key2 are keyword arguments,
args is a tuple holding a variable number of positional arguments,
and kwargs is a dictionary holding a variable number of
keyword arguments.
This document describes how to program with the
args and kwargs variables and why these are handy in many
situations.
Let us start by making a function that takes an arbitrary number of arguments and computes their sum:
>>> def add(*args):
... print 'args:', args
... s = 0
... for arg in args:
... s = s + arg
... return s
...
>>> add(1)
args: (1,)
1
>>> add(1,5,10)
args: (1, 5, 10)
16
We observe that args is a tuple and that all the arguments we
provide in a call to add are stored in args.
Combination of ordinary positional arguments and a variable number of
arguments is allowed, but the *args argument must appear after
the ordinary positional arguments, e.g.,
def f(pos1, pos2, pos3, *args):
In each call to f we must provide at least three arguments.
If more arguments are supplied in the call,
these are collected in the args
tuple inside the f function.
Consider a mathematical function with one independent variable
\( t \) and a parameter \( v_0 \), as in \( y(t;v_0) = v_0t - \frac{1}{2}gt^2 \).
A more general case with \( n \) parameters is
\( f(x; p_1,\ldots,p_n) \). The Python implementation
of such functions can take both the independent variable and the
parameters as arguments: y(t, v0) and f(x, p1, p2, ...,pn).
Suppose that we have
a general library routine
that operates on functions of one variable. The routine can, e.g.,
perform numerical differentiation, integration, or root finding. A simple
example is a numerical differentiation function
def diff(f, x, h):
return (f(x+h) - f(x))/h
This diff function cannot be used with functions f that
take more than one argument. For example, passing an y(t, v0) function
as f leads to the exception
TypeError: y() takes exactly 2 arguments (1 given)
A good solution to this problem is to make a class Y that
has a __call__(self, t) method and that stores \( v_0 \) as an attribute.
Here we shall describe an alternative solution that allows our
y(t, v0) function to be used as is.
The idea is that we pass additional arguments for the parameters in
the f function through the diff function.
That is, we view the f function as f(x, *f_prms) in diff.
Our diff routine can then be written as
def diff(f, x, h, *f_prms):
print 'x:', x, 'h:', h, 'f_prms:', f_prms
return (f(x+h, *f_prms) - f(x, *f_prms))/h
Before explaining this function in detail, we demonstrate that it works in an example:
def y(t, v0):
g = 9.81
return v0*t - 0.5*g*t**2
dydt = diff(y, 0.1, 1E-9, 3) # t=0.1, h=1E-9, v0=3
The output from the call to diff becomes
x: 0.1 h: 1e-09 f_prms: (3,)
The point is that the v0 parameter, which we want to pass on to
our y function, is now stored in f_prms.
Inside the diff function, calling
f(x, *f_prms)
is the same as if we had written
f(x, f_prms[0], f_prms[1], ...)
That is, *f_prms in a call takes all the values in
the tuple *f_prms and places them after each other as positional
arguments. In the present example with the y function,
f(x, *f_prms) implies
f(x, f_prms[0]), which for the current set of argument values
in our example
becomes a call y(0.1, 3).
For a function with many parameters,
def G(x, t, A, a, w):
return A*exp(-a*t)*sin(w*x)
the output from
dGdx = diff(G, 0.5, 1E-9, 0, 1, 0.6, 100)
becomes
x: 0.5 h: 1e-09 f_prms: (0, 1, 1.5, 100)
We pass here the arguments t, A, a, and w,
in that sequence, as the last four arguments to diff, and all
the values are stored in the f_prms tuple.
The diff function also works for a plain function f with
one argument:
from math import sin
mycos = diff(sin, 0, 1E-9)
In this case, *f_prms becomes an empty tuple, and
a call like f(x, *f_prms) is just f(x).
The use of a variable set of arguments for sending problem-specific
parameters through a general library function, as we have demonstrated
here with the diff function, is perhaps the most frequent use of
*args-type arguments.
A simple test function
>>> def test(**kwargs):
... print kwargs
exemplifies that kwargs is a dictionary inside the test
function, and that we can pass any set of keyword arguments to test,
e.g.,
>>> test(a=1, q=9, method='Newton')
{'a': 1, 'q': 9, 'method': 'Newton'}
We can combine an arbitrary set of positional and keyword arguments, provided all the keyword arguments appear at the end of the call:
>>> def test(*args, **kwargs):
... print args, kwargs
...
>>> test(1,3,5,4,a=1,b=2)
(1, 3, 5, 4) {'a': 1, 'b': 2}
From the output we understand that all the arguments in the call where
we provide a name and a value are treated as keyword arguments and
hence placed in kwargs, while all the remaining arguments are
positional and placed in args.
def y(t, v0=1):
g = 9.81
return v0*t - 0.5*g*t**2
In the diff function we transfer the parameters in the f
function as a set of keyword arguments **f_prms:
def diff(f, x, h=1E-10, **f_prms):
print 'x:', x, 'h:', h, 'f_prms:', f_prms
return (f(x+h, **f_prms) - f(x, **f_prms))/h
In general, the **f_prms argument in a call
f(x, **f_prms)
implies that all the key-value pairs in **f_prms are
provided as keyword arguments:
f(x, key1=f_prms[key1], key2=f_prms[key2], ...)
In our special case with the y function and the call
dydt = diff(y, 0.1, h=1E-9, v0=3)
f(x, **f_prms) becomes y(0.1, v0=3).
The output from diff is now
x: 0.1 h: 1e-09 f_prms: {'v0': 3}
showing explicitly that our v0=3 in the call to diff is placed
in the f_prms dictionary.
The G function from the section Variable number of positional arguments
can also have its parameters as keyword arguments:
def G(x, t=0, A=1, a=1, w=1):
return A*exp(-a*t)*sin(w*x)
We can now make the call
dGdx = diff(G, 0.5, h=1E-9, t=0, A=1, w=100, a=1.5)
and view the output from diff,
x: 0.5 h: 1e-09 f_prms: {'A': 1, 'a': 1.5, 't': 0, 'w': 100}
to see that all the parameters get stored in f_prms.
The h parameter can be placed anywhere in the collection
of keyword arguments, e.g.,
dGdx = diff(G, 0.5, t=0, A=1, w=100, a=1.5, h=1E-9)
We can allow the f function of one variable and a set of parameters
to have the general form f(x, *f_args, **f_kwargs). That is,
the parameters can either be positional or keyword arguments.
The diff function must take the arguments *f_args
and **f_kwargs and transfer these to f:
def diff(f, x, h=1E-10, *f_args, **f_kwargs):
print f_args, f_kwargs
return (f(x+h, *f_args, **f_kwargs) -
f(x, *f_args, **f_kwargs))/h
This diff function gives the writer of an f function full
freedom to choose positional and/or keyword arguments for the
parameters. Here is an example of the G function where we
let the t parameter be positional and the other parameters be
keyword arguments:
def G(x, t, A=1, a=1, w=1):
return A*exp(-a*t)*sin(w*x)
A call
dGdx = diff(G, 0.5, 1E-9, 0, A=1, w=100, a=1.5)
gives the output
(0,) {'A': 1, 'a': 1.5, 'w': 100}
showing that t is put in f_args and transferred as
positional argument to G, while A, a, and w
are put in f_kwargs and transferred as keyword arguments.
We remark that in the last call to diff, h and t
must be treated as positional arguments, i.e., we cannot write
h=1E-9 and t=0 unless all arguments in the call are on the
name=value form.
In the case we use both *f_args and
**f_kwargs arguments in f and there is no need for
these arguments, *f_args becomes an empty tuple and
**f_kwargs becomes an empty dictionary. The example
mycos = diff(sin, 0)
shows that the tuple and dictionary are indeed empty since diff
just prints out
() {}
Therefore, a variable set of positional and keyword arguments can
be incorporated in a general library function such as
diff without any disadvantage, just the benefit that diff
works with different types of f functions: parameters as global
variables, parameters as additional positional arguments,
parameters as additional keyword arguments, or parameters as instance
variables.
The program varargs1.py in the src/varargs
folder implements the examples in this document.
The term time has multiple meanings on a computer. The elapsed time or wall clock time is the same time as you can measure on a watch or wall clock, while CPU time is the amount of time the program keeps the central processing unit busy. The system time is the time spent on operating system tasks like I/O. The concept user time is the difference between the CPU and system times. If your computer is occupied by many concurrent processes, the CPU time of your program might be very different from the elapsed time.
Python has a time module with some useful functions for
measuring the elapsed time and the CPU time:
import time
e0 = time.time() # elapsed time since the epoch
c0 = time.clock() # total CPU time spent in the program so far
<do tasks...>
elapsed_time = time.time() - e0
cpu_time = time.clock() - c0
The term
epoch means initial time (time.time() would return 0), which
is 00:00:00 January 1, 1970.
The time module also has numerous functions for nice formatting
of dates and time, and the newer datetime module
has more functionality and an improved interface.
Although the timing has a finer resolution than seconds, one should
construct test cases that last some seconds to obtain reliable results.
To measure the efficiency of a certain set of statements, an
expression, or a function call,
the code should be run a large number of times so the
overall CPU time is of order seconds.
Python's timeit module has functionality for
running a code segment repeatedly. The simplest and most convenient
way of using timeit is within an IPython shell. Here is a
session comparing the efficiency of sin(1.2) versus math.sin(1.2):
In [1]: import math
In [2]: from math import sin
In [3]: %timeit sin(1.2)
10000000 loops, best of 3: 198 ns per loop
In [4]: %timeit math.sin(1.2)
1000000 loops, best of 3: 258 ns per loop
That is,
looking up sin through the math prefix degrades the performance by a
factor of \( 258/198\approx 1.3 \).
Any statement, including function calls, can be timed the same way.
Timing of multiple statements is possible by using %%timeit.
The timeit module can be used inside ordinary programs as demonstrated
in the file pow_eff.py.
Along with CPU time measurements it is often convenient to print out
information about the hardware on which the experiment was done.
Python has a module platform with information on the current
hardware. The function scitools.misc.hardware_info
applies the platform module and other modules
to extract relevant hardware information. A sample call is
>>> import scitools.misc, pprint
>>> pprint.pprint(scitools.misc.hardware_info())
{'numpy.distutils.cpuinfo.cpu.info': [
{'address sizes': '40 bits physical, 48 bits virtual',
'bogomips': '4598.87',
'cache size': '4096 KB',
'cache_alignment': '64',
'cpu MHz': '2299.435',
...
},
'platform module': {
'identifier': 'Linux-3.11.0-12-generic-x86_64-with-Ubuntu-13.10',
'python build': ('default', 'Sep 19 2013 13:48:49'),
'python version': '2.7.5+',
'uname': ('Linux', 'hpl-ubuntu2-mac11', '3.11.0-12-generic',
'#19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013',
'x86_64', 'x86_64')}}
}
A profiler computes the time spent in the various functions of a program. From the timings a ranked list of the most time-consuming functions can be created. This is an indispensable tool for detecting bottlenecks in the code, and you should always perform a profiling before spending time on code optimization. The golden rule is to first write an easy-to-understand program, then verify it, then profile it, and then think about optimization.
Premature optimization is the root of all evil.
Donald Knuth, computer scientist, 1938-.
Python 2.7 comes with two recommended
profilers, implemented in the modules cProfile
and profiles.
The section The Python Profilers in the Python Standard Library documentation
[12] has a good introduction to
the usage of these modules.
The results produced
by the modules are normally processed by a special statistics
utility pstats developed for analyzing profiling results.
The usage of the profile, cProfile,
and pstats modules is straightforward,
but somewhat tedious. The SciTools
package therefore comes with a command
scitools profiler that allows you to profile
any program (say) m.py by just writing
Terminal> scitools profiler m.py c1 c2 c3
Here, c1, c2, and c3 are command-line arguments
to m.py.
A sample output might read
1082 function calls (728 primitive calls) in 17.890 CPU seconds
Ordered by: internal time
List reduced from 210 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
5 5.850 1.170 5.850 1.170 m.py:43(loop1)
1 2.590 2.590 2.590 2.590 m.py:26(empty)
5 2.510 0.502 2.510 0.502 m.py:32(myfunc2)
5 2.490 0.498 2.490 0.498 m.py:37(init)
1 2.190 2.190 2.190 2.190 m.py:13(run1)
6 0.050 0.008 17.720 2.953 funcs.py:126(timer)
...
In this test, loop1 is the most expensive function, using 5.85
seconds, which is to be compared with 2.59 seconds for the next most
time-consuming function, empty. The tottime entry is the total
time spent in a specific function, while cumtime reflects the total
time spent in the function and all the functions it calls. We refer
to the documentation of the profiling tools in the Python Standard
Library documentation for detailed information on how to interpret the
output.
The CPU time of a Python program typically increases with a factor of
about five when run under the administration of the profile module.
Nevertheless, the relative CPU time among the functions are
not much affected by the profiler overhead.