Using Python in an HPC environment

Using Python in an HPC environment

This course aims to give a brief, but comprehensive introduction to using Python in an HPC environment. You will learn how to use modules to load Python, how to find site installed Python packages, as well as how to install packages yourself. In addition, you will learn how to use virtual environments, write a batch script for running Python, use Python in parallel, and how to use Python for ML and on GPUs. 

The course is a cooperation between UPPMAX (Rackham, Snowy, Bianca) and HPC2N (Kebnekaise) and will focus on the compute systems at both centres. For the site-specific part of the course you will be divided into groups depending on which center you will be running your code, as the approach is somewhat different. 

This course will consist of lectures interspersed with hands-on sessions where you get to try out what you have just learned.

Remote/online participation: The course will be completely online and we will use Zoom. More information about connecting and such will be sent to the participants close to the course.

The goal for the course is that you will be able to

• Load Python modules and site-installed Python packages
• Create a virtual environment
• Install Python packages with pip (Kebnekaise, Rackham, Snowy)
• Install Python packages with conda (Bianca)
• Write a batch script for running Python
• Use Python in parallel
• Use Python for ML
• Use GPUs with Python

Prerequisites: familiarity with the LINUX command line, basic Python

Preliminary Schedule

Friday, 2022-09-09

• 09:15  Introduction to Python on UPPMAX and HPC2N systems 
•            Loading modules and running Python codes 
•            Dealing with packages 
•            Creating isolated environments
•            Separated session for Kebnekaise/Bianca
• 12:00 - 13:15  Lunch
•            Batch scripts for Python jobs 
•            Parallelising a simple Python code
•            Using Python for Machine Learning jobs
•            Using GPUs with Python
•            Q&A on-demand and Summary 

Materials: Lectures and notes | Exercises | Recorded lectures and demos

Participation is free and open to all users from the SNIC HPC centres and academics throughout the Nordic region. 

Please make sure have an account at SUPR as well as at UPPMAX or HPC2N if you want to participate in the hands-on part of the training. There will be a course project that can be used to run the examples in during the hands-on. If you don't have an account at SUPR and/or UPPMAX/HPC2N, you will be contacted with further instructions for how to create those. You are encouraged to sign up to SUPR as soon as possible after registering for the course.

Course project: As part of the hands-on, you will be given temporary access to a course project, which will be used for running the hands-on examples. There are some policies regarding this, that we ask that you follow:

• You may be given access to the project before the course; please do not use the allocation for running your own codes in. Usage of the project before the course means the priority of jobs submitted to it goes down, diminishing the opportunity for you and your fellow participants to run the examples during the course. You can read more detailed information about the job policies at HPC2N here.
• The course project will be open 1-2 weeks after the course, giving the participants the opportunity to test run examples and shorter codes related to the course. During this time, we ask that you only use it for running course related jobs. Use your own discretion, but it could be: (modified) examples from the hands-on, short personal codes that have been modified to test things learned at the course, etc.
• Anyone found to be misusing the course project, using up large amounts of the allocation for their own production runs, will be removed from the course project.
• You will likely also be given access to a storage area connected to the compute project. Any data you store there should be course-related and if you wish to save it you should copy it to somewhere else soon after the course as it will be deleted about a month later.

The course uses computing resources provided by the Swedish National Infrastructure for Computing (SNIC) at HPC2N partially funded by the Swedish Research Council through grant agreement no. 2018-05973.

Time and date: 9 September 2022, 09:00-16:00

Location: ONLINE. Zoom link will be sent to participants a few days before the course.

Deadline for registration: 31 August 2022. REGISTRATION CLOSED

For registration, please go to the registration page.