Problem setting

Scripts for installing ready-made packages

We have developed a little tool where one can list the desired Debian or Python packages in a computing environment in a file with default name debkpg.txt. This file may also contain plain Unix commands for doing other types of installation, like pip install, or cloning of source code repositories with subsequent execution of a setup.py file. Concrete examples are listed below.

A little Python script deb2sh.py reads the installation specification in some file debpkg_minimal.txt and creates a Bash script install_minimal.sh and an equivalent Python script install_minimal.py for running all the necessary operating system commands to install all the packages in the correct order. The script aborts if any package cannot be installed successfully. The problem must then be fixed, or the package must in worst case be removed (just comment out the install line(s) in the Bash or Python script). The script can thereafter be rerun again.

The following is an extract of packages as they are listed in the mentioned debpkg_minimal.txt file:

# Minimal installation for a Python ecosystem
# for scientific computing

# Editors
emacs python-mode gedit vim ispell

# Compilers
gcc g++ gawk f2c gfortran
autoconf automake autotools-dev

# Numerical libraries
libatlas-base-dev libsuitesparse-dev

# Python
idle
python-pip
python-dev
# Matplotlib requires libfreetype-dev libpng-dev
# (otherwise pip install matplotlib does not work)
libfreetype6-dev libpng-dev
pip install numpy
pip install sympy
#pip install matplotlib   # pip may fail for matplotlib
python-matplotlib
pip install scipy

# ScientificPython must be installed from source
$ if [ ! -d srclib ]; then mkdir srclib; fi
$ cd srclib
$ hg clone https://bitbucket.org/khinsen/scientificpython
$ cd scientificpython
$ sudo python setup.py install
$ cd ../..

The syntax has four elements:

comment lines are just copied to the Bash and Python installation scripts,
lines starting with $ are plain Unix commands and run by the installation scripts,
lines starting with pip install lists packages to be installed with pip, while
all other non-blank lines are supposed to list the name of Debian packages to be installed by sudo apt-get install commands.

The examples above show all four line types. Observe in particular how we can freely add Unix commands to download ScientificPython from its Bitbucket repo (done in the srclib subdirectory) and install the package manually by running setup.py the usual way.

Some examples on lines in the automatically generated install_minimal.sh script are

#!/bin/bash
# Automatically generated script. Based on debpkg.txt.

function apt_install {
  sudo apt-get -y install $1
  if [ $? -ne 0 ]; then
    echo "could not install $1 - abort"
    exit 1
  fi
}

function pip_install {
  for p in $@; do
    sudo pip install $p
    if [ $? -ne 0 ]; then
      echo "could not install $p - abort"
      exit 1
    fi
  done
}

function unix_command {
  $@
  if [ $? -ne 0 ]; then
    echo "could not run $@ - abort"
    exit 1
  fi
}

sudo apt-get update --fix-missing

# Minimal installation for a Python ecosystem
# for scientific computing

# Editors
apt_install python-mode gedit vim ispell
...
pip_install numpy
pip_install sympy
apt_install scipy
...
# ScientificPython must be installed from source
unix_command if [ ! -d srclib ]; then mkdir srclib; fi
unix_command cd srclib
unix_command hg clone https://bitbucket.org/khinsen/scientificpython
unix_command cd scientificpython
unix_command sudo python setup.py install

Notice.

Installation commands may fail. Therefore we have made separate functions for doing the apt-get and pip install commands. We test the value of the environment variable $? after the installation of a package: a successful installation implies value of 0, while values different from 0 mean that something went wrong. We then abort the script with exit 1.
The apt-get install command will prompt the user for questions for every package, but here we use the option -y to automatically rely on default answers, i.e., accepting yes to all questions.

The corresponding lines in the equivalent, automatically generated install.py file look as follows.

import commands, sys

def system(cmd):
    """Run system command cmd."""
    failure, output = commands.getstatusoutput(cmd)
    if failure:
       print 'Command\n  %s\nfailed.' % cmd
       print output
       sys.exit(1)

system('sudo apt-get update --fix-missing')

system('sudo apt-get -y install python-mode gedit vim ispell')
...
system('pip install numpy')
system('pip install sympy')
system('sudo apt-get -y install scipy')
...
system('if [ ! -d srclib ]; then mkdir srclib; fi')
system('cd srclib')
sytem('hg clone https://bitbucket.org/khinsen/scientificpython')
system(' cd scientificpython')
system('sudo python setup.py install')

The Python script does not test the Unix environment variable $?, but the first return value from the getstatusoutput function acts as the value of $?.

We can use the Bash or Python script to easily automate installation of packages in the Vagrant machine. More powerful, industry standard tools for setting up complete software environments are Chef and Puppet.

Vagrant4scientists

Scripts for installing ready-made packages