Setting Up Programming Environment

In order to run hands-on exercises in this course, you need the Python package and several depenencies.

  • If you use your own computer to run exercises, you should follow the instructions described below to install relevant packages and setup specific programming environment before running hands-on exercises.

  • You can use an HPC cluster if you have access to one to run hands-on exercises. Below we provide a short description to login to the LUMI cluster, load the modules, and run interactive/batch jobs.

Local Installation

Install miniforge

If you already have a preferred way to manage Python versions and libraries, you can stick to that. If not, we recommend that you install Python3 and all libraries using Miniforge, a free minimal installer for the package, dependency and environment manager conda.

Please follow the installation instructions to install Miniforge.

Make sure that conda is correctly installed:

$ conda --version
conda 24.11.2

Install python programming environment on personal computer

The course environment includes the core scientific Python stack plus tools used throughout the course exercises:

  • Benchmarking and profiling: pyperf, line_profiler, snakeviz, py-spy

  • Optimization: numexpr, cython, numba, pythran, transonic

  • Parallelization: dask, distributed, mpi4py

  • Interactive work: jupyter, jupyterlab, ipympl, ipywidgets

The local environment files are available as environment.yml and requirements.txt.

For Mac users

With conda installed, install the required dependencies by running:

$ conda env create --yes -f content/env/environment.yml

This will create a new environment writing-performance-python which you need to activate by:

$ conda activate writing-performance-python

Ensure that the Python version is fairly recent:

$ python --version
Python 3.12.8

Finally, open Jupyter-Lab in your browser:

$ jupyter-lab

If you use VS Code, you can use the installed writing-performance-python programming environment by choosing Select Kernel in the upper right corner, then Python Environments, and then selecting the course environment.

For Linux users

With conda installed, run the same command from the repository root:

$ conda env create --yes -f content/env/environment.yml
$ conda activate writing-performance-python

If you prefer pip, create and activate a virtual environment, then install from requirements.txt:

$ python -m venv .venv
$ source .venv/bin/activate
$ python -m pip install -r content/env/requirements.txt

For Windows users

Install Miniforge and use the Miniforge Prompt from the repository root:

$ conda env create --yes -f content/env/environment.yml
$ conda activate writing-performance-python

If you prefer pip, create and activate a virtual environment in PowerShell:

$ py -m venv .venv
$ .venv\Scripts\Activate.ps1
$ py -m pip install -r content/env/requirements.txt

Using HPC Cluster

LUMI

Login to LUMI cluster

Follow practical instructions HERE to get your access to LUMI cluster.

  • On Step 5, you can login to LUMI cluster through terminal.

  • On Step 6, you can login to LUMI cluster from the web-interface.

Running jobs on LUMI cluster

If you want to run an interactive job asking for 1 node, 1 GPU, and 1 hour:

$ salloc -A project_XXXXX -N 1 -t 1:00:00 -p standard-g --gpus-per-node=1

$ srun <some-command>

Exit interactive allocation with exit.

You can also submit your job with a batch script submit.sh:

#!/bin/bash -l
#SBATCH --account=project_XXXXX
#SBATCH --job-name=example-job
#SBATCH --output=examplejob.o%j
#SBATCH --error=examplejob.e%j
#SBATCH --partition=standard-g
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=1:00:00

srun <some_command>

Some useful commands are listed below:

  • Submit the job: sbatch submit.sh

  • Monitor your job: squeue --me

  • Kill job: scancel <JOB_ID>

Using pyhpda programming environment on LUMI cluster

We have installed the pyhpda programming environment on LUMI. You can follow instructions below to activate it and login to LUMI cluster, either via terminal or through the web-interface.

Login to LUMI cluster via terminal and then the commands below to check and activate the pyhpda environment. You should get similar output as shown below.

$ /projappl/projecXXXXX10/miniconda3/bin/conda init

$ source ~/.bashrc

$ which conda
/project/project_XXXXX/miniconda3/condabin/conda

$ conda activate pyhpda

$ which python
/project/project_XXXXX/miniconda3/envs/pyhpda/bin/python

Login to LUMI cluster via the web interface and then select Jupyter (not Jupyter for courses) icon for an interactive session, and provide the following values in the form to launch the jupyter lab app.

  • Project: project_XXXXX

  • Partition: interactive

  • Number of CPU cores: 2

  • Time: 4:00:00

  • Working directory: /projappl/project_XXXXX

  • Python: Custom

  • Path to python: /project/project_XXXXX/miniconda3/envs/pyhpda/bin/python

  • check for Enable system installed packages on venv creation

  • check for Enable packages under ~/.local/lib on venv start

  • Click the Launch button, wait for minutes until your requested session was created.

  • Click the Connect to Jupyter button, and then select the Python kernel Python 3 (venv) for the created Jupyter notebooks.