Using Python in an HPC environment, 5-6 December 2024

Using Python in an HPC environment

This two-days online course, taking place on 5-6 December 2024, 09:00-17:00 each day, aims to give a brief, but comprehensive introduction to using Python in an HPC environment.

The first day you will learn how to use modules to load Python, how to find site installed Python packages, as well as how to install packages yourself. In addition, you will learn how to use virtual environments, write a batch script for running Python, and get an intro to how to use Python on GPUs. The second day we will look at using Pandas, parallel Python, big data, and also ML/DL with Python.

The course is a cooperation between UPPMAX, HPC2N, and LUNARC. The instructors will mainly use UPPMAX's systems for demos and there will be hands-on exercises for the participants.

This course will consist of lectures interspersed with hands-on sessions where you get to try out what you have just learned.

Remote/online participation: The course will be completely online and we will use Zoom. More information about connecting and such will be sent to the participants close to the course.

The goal for the course is that you will be able to

• Load Python modules and site-installed Python packages
• Create a virtual environment
• Install Python packages with pip and/or conda
• Write a batch script for running Python
• Use Python in parallel
• Use Python for ML
• Use GPUs with Python
• Use Pandas
• Python and big data

Prerequisites: familiarity with the LINUX command line, basic Python

Preliminary Schedule

Thursday, 2024-12-05, 9:00-17:00

• 09:00  Introduction to Python. Logging in. Cluster info. Load/run Python. Getting exercises (tarball). Editors
• 10:15 Dealing with system installed packages. Creating isolated environments and installing extra Python packages
• 11:15 Basic Slurm
  • Batch scripts for Python jobs 
  • Interactive jobs, Jupyter, Spyder
  • Allocating GPUs
• 12:00 - 13:00  Lunch
• 13:00 Analysis with Python
  • Using IDE work environments
    • Jupyter/Spyder/VScode?
  • matplotlib
• 14:30 Using GPUs
• 15:30 Use cases and Q&A
• 16:35 Ending with evaluation

Friday, 2024-12-06, 9:00-17:00

• 9:00 Analysis with Python
  • Pandas
  • Seaborn
• 11:00 Parallelism
  • MPI
  • Dask
  • Processes
• 13:15 Big data with Python
  • csv
  • xarray/netcdf/hdf5 ?
  • Chunking (dask+pandas?)
• 14:15 ML + DL
  • pytorch
  • tensorflow
  • sklearn
  • ...
• 16:00 Use cases and Q&A
• 16:45 Ending with evaluation

Materials: Lectures and notes | Exercises | Recorded lectures and demos

Participation is free and open to all users within NAISS, whether their site has NAISS resources or not.

Please make sure you have an account at SUPR if you want to participate in the hands-on part of the training. There will be a course project that can be used to run the examples in during the hands-ons, if you do not already have an account with access. If you do not have an account at SUPR and UPPMAX, you will be contacted with further instructions for how to create those. You are encouraged to sign up to SUPR as soon as possible after registering for the course.

Course project: As part of the hands-on, you will be given temporary access to a course project at some of UPPMAX's systems, which will be used for running the hands-on examples. There are some policies regarding this, that we ask that you follow:

• You may be given access to the project before the course; please do not use the allocation for running your own codes in. Usage of the project before the course means the priority of jobs submitted to it goes down, diminishing the opportunity for you and your fellow participants to run the examples during the course.
• The course project will be open 1-2 weeks after the course, giving the participants the opportunity to test run examples and shorter codes related to the course. During this time, we ask that you only use it for running course related jobs. Use your own discretion, but it could be: (modified) examples from the hands-on, short personal codes that have been modified to test things learned at the course, etc.
• Anyone found to be misusing the course project, using up large amounts of the allocation for their own production runs, will be removed from the course project.
• You will likely also be given access to a storage area connected to the compute project. Any data you store there should be course-related and if you wish to save it you should copy it to somewhere else soon after the course as it will be deleted about a month later.

The course uses computing resources provided by the National Academic Infrastructure for Super­computing in Sweden (NAISS) at UPPMAX, partially funded by the Swedish Research Council through grant agreement no. 2022-06725.

If you are affiliated with UmU, SLU, LTU, MIUN, or IRF you can use HPC2N's systems instead of UPPMAX. If you are affiliated with LU, you can use LUNARC's resources.

Time and date: 5-6 December 2024, 09:00-17:00 each day

On-boarding: 3 December 2024, 13:00-14:00 (get optional help with final preparations)

Location: ONLINE. Zoom link will be sent to participants a few days before the course.

Instructors: Rebeccas Pitts, Pedro Ojeda-May, Jayant Yadav, Björn Claremar, Birgitte Brydsö

Deadline for registration: 2024-11-26

Please register by filling in the below form. Fields marked with a * are mandatory.  REGISTRATION CLOSED!