system news

Since the current operating system version, Ubuntu Xenial 16.04, that we are running on our clusters is reaching its End-Of-Life on 2021-04-30, we are upgrading to Ubuntu Focal 20.04.

The upgrade process will start 2021-04-19 and will be done with minimal impact to users running jobs.

Access to user data will not be affected by this upgrade.

There is already now a test environment available for users to check this out.

It is imperative that users test this out as soon as possible and notify us of any missing softwares.

We are noticing intermittent file server crashes causing problems. This causes problems with the file systems for $HOME and project storage. As a user it is mostly noticed by logins getting stuck after authentication and/or really slow filesystem access (simple ls might takes minutes).

The exact cause is not known but we are doing active debugging together with our vendor. During the debugging we might block the starting of new batch jobs. You can still submit jobs but they will not start.

Dear users,

During this maintenance, 2021-03-22 - 2021-03-25, we’re going to do some upgrades on the parallel file system, where home directories and project storage is located, along with other upgrades on Kebnekaise itself.

Since this maintenance affects the parallel file system we have to drain the batch nodes from running jobs. Login sessions will be disabled and active sessions will be terminated, during that period.

2021-02-11 Storage system issues causes Kebnekaise login and fileaccess problems

Update: The problems with the storage system was resolved 11:30 and the Kebnekaise login nodes are available again.

Migrating away from /pfs/nobackup at HPC2N

Dear PIs and users at HPC2N.

As you hopefully already know our new storage system is in full production since November.

There is still some work to be done by You, the user, to make the transition to Project Storage complete.
All data in your /pfs/nobackup$HOME space must be moved to a Project Storage directory or to your $HOME space depending on the type and amount of data.

We are currently having issues with the CVMFS subsystem on all HPC2N machines. This affects local software and modules, amongst other things.

Fixing is in progress, but might take a while before everything is sorted out.

We apologize for the inconvenience.

2021-01-13 11:59

The problem has been resolved

As a side effect of the recent file system upgrade we are observing a small set of user-installed programs crashing with a bus error when loading dynamic libraries from the PFS file system. We are working with the vendor to find the root cause of these and are running some bulk operations on the file system to mitigate the problem.

The problem is elusive and may only affect a particular set of nodes and may disappear when hashing or otherwise fully reading the affected files on that node, or reinstalling the affected files in another location.

We are currently experiencing filesystem problems on the Kebnekaise cluster and logins will likely fail or hang indefinitely.

The problem is under investigation.

Update 2020-12-09 12:00
The filesystem on the login nodes is now working.

We're currently doing some minor maintenance on the storage file system.

The batch queues on Kebnekaise are stopped, therefor no new jobs will be allowed to start.

Already running jobs will continue to run.

The file system should be available most of the time but may at times access may be stalled for shorter periods.

Logins may be slower than normal.

* UPDATE 2020-12-01 11:30 *

Maintenance done and batch queues now running again.

We have a maintenance window starting 2020-11-12 08:00 for migrating to our new storage system.

This affects batch queues and login nodes for all users.

Background

We have acquired a new storage system which will replace our old center storage.
The new storage is twice as large and will provide better performance overall for both user and project storage.

The old storage will be decommissioned and the data must therefore be migrated to the new system.