HPC2N
High Performance Computing Center North
2019-04-04:
We are experiencing severe slowdown on the /pfs/nobackup file system, affecting all accesses including running jobs.
This is caused by components in the storage system restarting for unknown reasons, investigation is ongoing.
*UPDATE* In order to identify what is going on we are forced to shut down the file system occasionally. The vendor is assisting in identifying and fixing the issue.
2019-03-29:
Normal power routing restored to all nodes.
2019-03-28:
Repairs completed, switchboard powered up.
2019-03-25:
Due to component delivery delays the final steps of repair are postponed. The new date for completion is Wednesday 2019-03-27.
2019-03-20:
Replacement parts and cables are en route. We currently estimate installation and recertification of the switchboard to be finished by the end of Monday 2019-03-25.
2019-03-15:
There was a (25-minute) power outage at Umeå University campus just before 09:30. This brought down our clusters (Kebnekaise and Abisko and killed all running jobs. It also has affected the Kebnekaise login nodes.
Power is back now, and we are in the process of taking up the clusters. We will add more information when we know what happened and/or when the clusters are back up.
*Update* The reason for the power outage was a severed cable that triggered a circuit breaker in one of the universities internal power stations.
Due to hardware failures in one of the storage servers we currently have reduced performance on the pfs file system.
The effect of this is that any file system operation may be slower or nonresponsive at times.
We are working on fixing the problem.
*PROBLEM FIXED*
On Monday 2019-03-04 09:00 we have a maintenance window to replace parts of the /pfs/nobackup file system.
We expect it to take a couple of hours.
All clusters will be affected, and jobs that have a requested timelimit that reaches beyond that point in time will not be allowed to start until after the service.
At around 19:20 today (30/1), the power to the clusters was lost. That means both kebnekaise and abisko is not running any jobs. All running jobs has has stopped. We are working on getting back the power and the cluster up. This message will be updates has we know more.
Login to the kebnekaise access nodes will stall until the kebnekaise cluster is back up. Abisko login node should work.
*Update* 2019-01-30 22:40: Abisko should be up and running again. Kebnekaise has some problems with /pfs and is still down. Probably will not be fixed until tomorrow.
We have an electrical maintenance planned in our cluster room 20190128 07:30 - 18:00
The work requires the power to be turned off and thus all clusters will need to be drained and taken down.
We have therefor set a reservation on the clusters preventing any jobs from running during that time frame.
We are currently experiencing a stability problem with the /pfs/nobackup file system.
We're fixing it as fast as we can.
*UPDATE 2019-01-03 18:17*
The stability problem should now be solved.
We suffered a major power failure once again.
The clusters are down and all jobs that was running are lost.
We're still waiting for power to come back.
*UPDATE 20181126 15:40*
Both Abisko and Kebnekaise are now up and running again.
We just suffered an almost city wide power failure.
All clusters are down and all jobs that where running has been lost.
We will bring things up as soon as we can when power comes back.
*UPDATE 20181112 17:40*
The power is back and the systems are back online.
