Python Environment

In order to run python applications using slurm, a virtual environment should be used to install the pip dependencies.

Set up the virtual environment

Warning

A virtual environment is specific to a machine and must be set up on each of the target nodes

First, connect to the node in which you want to install the virtual environment:

srun -p [partition] --pty bash -i

Note that you may need to setup the https proxy in order to allow pip to install packages:

export http_proxy=http://asim.lip6.fr:3128
export https_proxy=http://asim.lip6.fr:3128

Then, create a virtual environment at the current location:

python3 -m venv myproject_env

Activate the virtual environment:

source myproject_env/bin/activate

Now you are in the virtual environment, you can install the dependencies of your application, ussually the procedure is the following but refer to the README of the application.

cd path/app
python3 -m pip install -r requirements.txt

After that, you can leave the virtual environment:

deactivate

And finally leave the interactive session by pressing CTRL+D.

Info

Once you are confident that your install is working using the previous steps, you can repeat the process by creating a sbatch script with the commands above.

Use the virtual environment in your slurm job

Example slurm job to call app.py:

#!/bin/bash

source full_path/myproject_env/bin/activate
python3 app.py
deactivate