Cluster upgrade to Ubuntu Focal 20.04

This is a description of the upgrade to Ubuntu Focal 20.04 of our cluster and what users may have to do to take advantage of it.

When does it start?

The upgrade will start 2021-04-19.

When is it done?

The upgrade was finished 2021-04-27.

Why are we doing this?

The current operating system version we are using is Ubuntu Xenial 16.04 which reaches End-Of-Life 2021-04-30. We therefore have to upgrade to a newer version which will continue to receive (security)upgrades.

How the upgrade is done

To minimize the downtime and impact to the users, the upgrade will be done by

• disabling job submission to the old part of the cluster,
• moving batch nodes over to the Focal installation,
• upgrading the normal login nodes to Focal,
• redirecting submission of new jobs over to the Focal based nodes.

Jobs submitted before the upgrade starts will continue to run on the old part of the system (which will slowly shrink in size as nodes are upgraded).

What about user data?

Access to user data will not be affected in any way due to this upgrade.

Software, both pre-installed and user built

The upgrade also means that all the software currently available through the module system will be rebuilt. User built software likely also has to be rebuilt, especially if previously built using an older compiler version. This should not be done while jobs are running with the old build.

Note: this is likely to include any python virtual environments created.

Due to us rebuilding all pre-installed software there may be parts missing initially.

Testing the new Focal installation

We invite all users to start checking out what we have already prepared and let us know what other softwares and/or versions are required from the module system.
To do this we have prepared one node with the Focal installation that you can log in to and see what is currently built and what is missing. If you have built your own software please contact support using the information below before trying to use it on the Focal nodes, we may not have built the required toolchain or libraries yet.

The temporary login node to use for this pre-upgrade testing is kebnekaise-focal.hpc2n.umu.se.

It is already possible to submit jobs to a few upgraded nodes, both normal and GPU enabled. There are special partitions set up to handle the Focal part of the systems. When logged in to kebnekaise-focal the default is to send jobs to the Focal based partitions, please avoid explicitly targeting a partition (with the #SBATCH -p option) during the initial test phase. The Focal partitions are currently hidden, so to see jobs submitted there add "-a" to squeue, i.e.,

squeue -a -u your-user-name

Current status of the upgrade

Wednesday 2021-04-21

• We have now disabled logins on kebnekaise.hpc2n.umu.se (b-an01) and will be upgrading it during the day.
• We will disable submission of new jobs to the old part of the system during the day, the "largemem" and "knl" partitions are already disabled.
• We have a temporary thinlinc node with Focal available for testing, b-an02.hpc2n.umu.se. It does however not have a graphics card so not all functionality is available.
• Submission of jobs to the old part of the system will be disabled 12:30 CEST.

Wednesday 2021-04-21 14:00

• The normal login node, kebnekaise.hpc2n.umu.se, is now upgraded and available for use.
• Submission of jobs to the old part of the system have been disabled.

Thursday 2021-04-22 09:00

• The thinlinc nodes are now upgraded.
• The KNL partition is fully upgraded and usable.

Friday 2021-04-23 08:30

• The Largemem parition is now upgraded and usable. For the time being one must do "unset SBATCH_PARTITION" before trying to submit to the largemem partition.
• The temporary kebnekaise-focal node is now deleted. Please use the normal login nodes.

Tuesday 2021-04-27 09:30

• The upgrade can now be considered done, once the remaining jobs running in the old setup are finished those nodes will be upgraded.
• New logins will be able to run interactive jobs as usual and submit to Largemem and KNL partitions in the usual way. I.e. the SBATCH_PARTITION is no longer set in the environment.

Reporting problems, missing software etc

Please let us know what else you need by using the support form on SUPR or by mailing the information to support@hpc2n.umu.se.

The sooner you let us know what is missing, the sooner we can build it.

Yours sincerely, the sysadmins @ HPC2N