Python

Python is a powerful, dynamic, object-oriented programming language.

Python serves many purposes. It’s an operating system component, a standalone scripting/programming platform, and is used as middleware that enables other software to run. There are multiple versions of python and related software available at PSC. This page is designed to help figure out the right one for your project.

On Bridges-2, python is available:

• As part of the AI modules. The AI modules bundle AI software into a cohesive package, with attention paid to ensuring that the included packages work together seamlessly. The packages include pytorch, keras, tensorflow, and others. These modules are intended for use on Bridge-2’s GPU nodes.

See the AI module documentation for more information.

• As part of the Anaconda modules.  The Anaconda library includes a familiar collection of python modules and uses the conda packaging system, as well as conveniences such as the ipython shell.  The Anaconda2 module uses python2; the Anaconda3 module uses python3. For a more full featured version of python you may find the anaconda3 module helpful.

It is possible to install additional modules using the conda installer in a virtual environment without requiring additional privileges. It is recommended to install these packages in the user’s Ocean space as they can become quite large.

See the Anaconda documentation for more information.

• As a standalone python module. The general python module can be used for basic python scripting with a minimal environment. This python module may conflict with the AI and Anaconda modules, or other modules which load or use python.

It is possible to add desired python modules to your environment in your $HOME/.local directory, but these can become quite large so it is advised to first link the appropriate directory to your $PROJECT space in Ocean. This must be done for each version of Python the user wishes to access.

The following example works for python 3.8.x (the default as of the time of writing):

mkdir $PROJECT/.python
mkdir -p ~/.local/lib/python-3.8
mv ~/.local/lib/python-3.8  $PROJECT/.python
ln -s $PROJECT/.python  ~/.local/lib/python-3.8

Then to install a package, using numpy as an example:

python -m ensure-pip
python -m pip install numpy
python
>>> import numpy

See Usage on Bridges-2 (below) for more information.

• As part of the OS:  Python is a standard part of the OS on the compute nodes. Both  python2 and python3 are available. It is NOT recommended to rely on this as it is not guaranteed to remain accessible across OS upgrades. It is mentioned primarily as a caution.

Usage on Bridges-2

To see what versions of python are available and if there is more than one, which is the default, along with some help, type

module spider python

To use python, include a command like this in your batch script or interactive session to load the python module: (note ‘module load’ is case-sensitive):

module load python

Add-on packages

Various add-on packages such as numpy, scipy, and the MKL libraries are installed with individual versions. You can see what is available in a given version by first loading the module and then using the pip command:

module load python/x.x.x 
pip freeze            ##  for python 2
pip3 freeze          ## for python 3

This command will give the version number of the installed packges.

User installed add-on packages

You may also install add-on packages in your own directories.  After loading the appropriate python module, use the pip install command:

 pip install --user package-name=version-number

If you want to install a specific version of your package, add the optional version-number.