Using Python in an HPC environment

Using Python in an HPC environment

This course aims to give a brief, but comprehensive introduction to using Python in an HPC environment. You will learn how to use modules to load Python, how to find site installed Python packages, as well as how to install packages yourself. In addition, you will learn how to use virtual environments, write a batch script for running Python, use Python in parallel, and how to use Python for ML and on GPUs. 

The course is a cooperation between UPPMAX and HPC2N. The instructors will use UPPMAX's systems for demos and there will be hands-on exercises for the participants.

This course will consist of lectures interspersed with hands-on sessions where you get to try out what you have just learned.

Remote/online participation: The course will be completely online and we will use Zoom. More information about connecting and such will be sent to the participants close to the course.

The goal for the course is that you will be able to

• Load Python modules and site-installed Python packages
• Create a virtual environment
• Install Python packages with pip and/or conda
• Write a batch script for running Python
• Use Python in parallel
• Use Python for ML
• Use GPUs with Python

Prerequisites: familiarity with the LINUX command line, basic Python

Preliminary Schedule

Friday, 2023-12-01, 9:00-17:00 (16:00 - 17:00 optional helpdesk "bring your own code")

• 09:00  Introduction to Python. Logging in. Cluster info.
•            Loading modules and running Python codes 
•            Dealing with system installed packages 
•            Creating isolated environments and installing extra Python packages
•            Batch scripts for Python jobs 
•            Interactive jobs and Jupyter
• 12:00 - 13:00  Lunch
•            Parallelising a simple Python code
•            Using GPUs with Python
•            Using Python for Machine Learning jobs
•            Q&A and Summary 
• 16:00 - 17:00
•            Optional helpdesk - bring your own code

Materials: Lectures and notes | Exercises | Recorded lectures and demos

Participation is free and open to all users within NAISS, whether their site has NAISS resources or not.

Please make sure you have an account at SUPR if you want to participate in the hands-on part of the training. There will be a course project that can be used to run the examples in during the hands-ons, if you do not already have an account with access. If you don't have an account at SUPR and UPPMAX, you will be contacted with further instructions for how to create those. You are encouraged to sign up to SUPR as soon as possible after registering for the course.

Course project: As part of the hands-on, you will be given temporary access to a course project at some of UPPMAX's systems, which will be used for running the hands-on examples. There are some policies regarding this, that we ask that you follow:

• You may be given access to the project before the course; please do not use the allocation for running your own codes in. Usage of the project before the course means the priority of jobs submitted to it goes down, diminishing the opportunity for you and your fellow participants to run the examples during the course.
• The course project will be open 1-2 weeks after the course, giving the participants the opportunity to test run examples and shorter codes related to the course. During this time, we ask that you only use it for running course related jobs. Use your own discretion, but it could be: (modified) examples from the hands-on, short personal codes that have been modified to test things learned at the course, etc.
• Anyone found to be misusing the course project, using up large amounts of the allocation for their own production runs, will be removed from the course project.
• You will likely also be given access to a storage area connected to the compute project. Any data you store there should be course-related and if you wish to save it you should copy it to somewhere else soon after the course as it will be deleted about a month later.

The course uses computing resources provided by the National Academic Infrastructure for Super­computing in Sweden (NAISS) at UPPMAX, partially funded by the Swedish Research Council through grant agreement no. 2022-06725.

Time and date: 01 December 2023, 09:00-17:00

On-boarding: Wednesday, 29 November 2023, 13:30-14:30 (get help with final preparations)

Location: ONLINE. Zoom link will be sent to participants a few days before the course.

Deadline for registration: 2023-11-23

Please register by filling in the below form. Fields marked with a * are mandatory. REGISTRATION CLOSED!