Slurm MPI + OpenMP examples

Slurm MPI + OpenMP examples

This example shows a hybrid MPI/OpenMP job with 4 tasks and 48 cores per task, on Abisko.

#!/bin/bash
# Example with 4 tasks and 48 cores per task for MPI+OpenMP
#
# Project/Account
#SBATCH -A hpc2n-1234-56
#
# Number of MPI tasks
#SBATCH -n 4
#
# Number of cores per task
#SBATCH -c 48
#
# Runtime of this jobs is less then 12 hours.
#SBATCH --time=12:00:00
#

# Set OMP_NUM_THREADS to the same value as -c
# with a fallback in case it isn't set.
# SLURM_CPUS_PER_TASK is set to the value of -c, but only if -c is explicitly set
if [ -n "$SLURM_CPUS_PER_TASK" ]; then
  omp_threads=$SLURM_CPUS_PER_TASK
else
  omp_threads=1
fi
export OMP_NUM_THREADS=$omp_threads

srun ./mpi_openmp_program

# End of submit file

Sometimes it is possible to achieve better performance if you use 8 MPI tasks per node and 6 cores per task due to memory locality. This is on Abisko.

#!/bin/bash
# Example with 8x6 cores for MPI+OpenMP.
#
# Project/Account
#SBATCH -A hpc2n-1234-56
#
# Number of MPI tasks
#SBATCH -n 8
#
# Number of cores per task
#SBATCH -c 6
#
# Number of tasks per node
#SBATCH --ntasks-per-node=8
#
# Runtime of this jobs is less then 12 hours.
#SBATCH --time=12:00:00
#

# Set OMP_NUM_THREADS to the same value as -c
# with a fallback in case it isn't set.
# SLURM_CPUS_PER_TASK is set to the value of -c, but only if -c is explicitly set
if [ -n "$SLURM_CPUS_PER_TASK" ]; then
  omp_threads=$SLURM_CPUS_PER_TASK
else
  omp_threads=1
fi
export OMP_NUM_THREADS=$omp_threads


# --cpu_bind=rank_ldom is only possible if a complete node is
# allocated.
# This allocates one MPI task with its 6 OpenMP cores
# per NUMA unit.
srun --cpu_bind=rank_ldom ./mpi_openmp_program

# End of submit file

If your job needs more tasks, change number of tasks (-n) and (maybe) --tasks-per-node.

#!/bin/bash
# Example with 16x6 cores for MPI+OpenMP.
#
# Project/Account
#SBATCH -A hpc2n-1234-56
#
# Number of MPI tasks
#SBATCH -n 16
#
# Number of cores per task
#SBATCH -c 6
#
# Number of tasks per node
#SBATCH --ntasks-per-node=8
#
# Runtime of this jobs is less then 12 hours.
#SBATCH --time=12:00:00
#

# Set OMP_NUM_THREADS to the same value as -c
# with a fallback in case it isn't set.
# SLURM_CPUS_PER_TASK is set to the value of -c, but only if -c is explicitly set
if [ -n "$SLURM_CPUS_PER_TASK" ]; then
  omp_threads=$SLURM_CPUS_PER_TASK
else
  omp_threads=1
fi
export OMP_NUM_THREADS=$omp_threads

# --cpu_bind=rank_ldom is only possible if a complete node is
# allocated.
# This allocates one MPI task with its 6 OpenMP cores
# per NUMA unit.
srun --cpu_bind=rank_ldom ./mpi_openmp_program

# End of submit file

Slurm will allocate space for --tasks-per-node tasks on every node. In the example above we have 8 tasks per node and 16 tasks. (If you change the number of ntasks-per-node to 9 the jobs will not fit inside one Abisko node because 9 * 6 > 48 cores.) Kebnekaise has 28 cores per node.
More nodes will be allocated because the number of tasks requested is larger than the number of tasks-per-node, in this case one more node (16/8 = 2).

This example shows a hybrid MPI/OpenMP job with 4 tasks and 28 cores per task, on Kebnekaise. Note that Kebnekaise has 28 cores per node.

#!/bin/bash
# Example with 4 tasks and 28 cores per task for MPI+OpenMP
#
# Project/Account
#SBATCH -A hpc2n-1234-56
#
# Number of MPI tasks
#SBATCH -n 4
#
# Number of cores per task
#SBATCH -c 28
#
# Runtime of this jobs is less then 12 hours.
#SBATCH --time=12:00:00
#

# Set OMP_NUM_THREADS to the same value as -c
# with a fallback in case it isn't set.
# SLURM_CPUS_PER_TASK is set to the value of -c, but only if -c is explicitly set
if [ -n "$SLURM_CPUS_PER_TASK" ]; then
  omp_threads=$SLURM_CPUS_PER_TASK
else
  omp_threads=1
fi
export OMP_NUM_THREADS=$omp_threads

# On Kebnekaise you use mpirun instead of srun 
mpirun ./mpi_openmp_program

# End of submit file

You can submit scripts like those above into slurm with:

sbatch submit_script.sh

 

Updated: 2017-08-02, 15:47