Abisko - Libraries

Abisko - Libraries

[ Provided Libraries | MPI Libraries | Math Libraries and linkage | ACML libraries and linking | Intel MKL Libraries |Linking with MKL ]

Some of the libraries are already in the path. For the rest, you need to load the specific module first. The libraries on HPC2N systems includes parallel communications libraries as well as mathematical libraries. More detailed documentation on the libraries available follows below.

The examples below will load the default versions. To see all the versions available, type the command

module avail <module name>

where module name is the library name (e.g. openblas, lapack, libfftw, etc.).

MPI Libraries

The available versions of MPI libraries depend on the machine you are logged in to. All have OpenMPI and Intel MPI installed. The available MPI libraries are installed for all the provided compilers and can be seen by typing module avail and looking for e.g. openmpi or impi. Information about linking to the MPI libraries can be found on the Provided Compilers page.

OpenMPI

To compile

module load openmpi/<compiler>

and compile with

Language Command
Fortran77 mpif77
Fortran90 mpif90
Fortran95 N/A
C mpicc
C++ mpiCC

To run
Add this to your batchfile

module load openmpi/<compiler>
mpiexec <program>

Intel MPI

To compile

module load impi

Note that you also want to load a compiler module unless you want to compile with the default GNU compilers. In addition, if you want to use another version of impi, check which ones are available with

module avail impi

This is how you compile

Language Compile using GNU compilers Compile using Intel Compile using PathScale
Fortran77 mpif77 mpiifort mpifc -fc=pathf90
Fortran90 mpif90 mpiifort mpifc -fc=pathf90
Fortran95 N/A mpiifort mpifc -fc=pathf95
C mpicc mpiicc mpicc -cc=pathcc
C++ mpicxx mpiicpc mpicxx -cxx=pathCC

Add this to your batchfile

module load <compiler>
module load impi
srun <program>

Here are a few links to pages with more information about the different implementations of the MPI libraries.

OpenMPI

Intel MPI

Math Libraries

The following list is not exhaustive, but it covers the most 'popular' of the libraries that are installed on Abisko at HPC2N. There are more detailed examples of using the math libraries here.

Blas

Blas is available in the form of OpenBLAS, Intel MKL and ACML. Intel MKL is often recommended if you are compiling with Intel compilers. The best way to access it is one of these:

OpenBLAS:

$ module load openblas/<compiler>

To compile and link your program use (remember to also load the compiler you wish to use):

Example (Pathscale, Fortran program, non-threaded version)

pathf90 blas_test.f90 $OPENBLAS_LDFLAGS $OPENBLAS_LIBS

Example (Pathscale, Fortran program, threaded version)

pathf90 blas_test.f90 $OPENBLAS_LDFLAGS $OPENBLAS_MP_LIBS -lopenmp -pthread

To see alternative versions, do a module avail and look for openblas.

All BLAS versions:
The number of threads is by default set to 1. To change it use

Bourne (bash, sh) and Korn:

export OMP_NUM_THREADS=6

C-shell family (csh, tcsh):

setenv OMP_NUM_THREADS 6

BLAS from ACML

You can also use the BLAS routines included with the ACML libraries. First you must load the module:

$ module load acml/<compiler>

To compile and link your program use (remember to also load the compiler you wish to use):

Example, (Pathscale, Fortran program, non-threaded version)

pathf90 program.f90 $ACML_LDFLAGS $ACML_LIBS

Example, (Pathscale, Fortran program, threaded version)

pathf90 program.f90 $ACML_MP_LDFLAGS $ACML_MP_LIBS -lopenmp -pthread

To see alternative versions, do a module avail and look for acml. For information about usage in your makefile, see module help acml/<compiler> after loading the module.

LAPACK

LAPACK is installed on all systems. The 3.2.1 and later versions have a couple of functions rewritten for enhanced speed and thread-safety compared to the versions available prior to 3.1. Three sets of functions in the older lapack, xLACON, xLASQ3 and xLASQ4, are not thread-safe and new versions are available. In lapack they are called xLACN2, xLAZQ3 and xLAZQ4. See http://www.netlib.org/lapack/lapack-3.2.html and http://www.netlib.org/lapack/lawnspdf/lawn158.pdf for further information.

LAPACK is written in Fortran77 and provides routines for solving systems of simultaneous linear equations, least-squares solutions of linear systems of equations, eigenvalue problems, and singular value problems. The associated matrix factorizations (LU, Cholesky, QR, SVD, Schur, generalized Schur) are also provided, as are related computations such as reordering of the Schur factorizations and estimating condition numbers. Dense and banded matrices are handled, but not general sparse matrices. In all areas, similar functionality is provided for real and complex matrices, in both single and double precision.

You can also use the Intel MKL or ACML library version of LAPACK.

To use the Fortran based lapack library you must first load its module and one BLAS module, as well as the compiler you wish to use.

$ module load lapack/<compiler>/<version>
$ module load openblas/<compiler> 

After that, link with LAPACK like this

Lapack (example, Fortran program, PathScale compilers):

$ pathf90 program.f90 -o program $LAPACK_LDFLAGS $OPENBLAS_LDFLAGS $LAPACK_LIBS $OPENBLAS_LIBS

Lapack (example, C program, PathScale compilers):

$ pathcc program.c -o program $LAPACK_LDFLAGS $OPENBLAS_LDFLAGS $LAPACK_LIBS $OPENBLAS_LIBS -lpathfortran

Different versions exist for the various compilers. Do a module avail and look for 'lapack'.

LAPACK from ACML

You can also use the LAPACK (and BLAS) included with the ACML libraries. First you must load the module:

$ module load acml/<compiler>

To compile and link your program use (remember to also load the compiler you wish to use):

Example, (Pathscale, Fortran program, non-threaded version)

pathf90 program.f90 $ACML_LDFLAGS $ACML_LIBS

Example, (Pathscale, Fortran program, threaded version)

pathf90 program.f90 $ACML_MP_LDFLAGS $ACML_MP_LIBS -lopenmp

To see alternative versions, do a module avail and look for acml. For information about usage in your makefile, see module help acml/<compiler> after loading the module.

BLACS

As of ScaLAPACK version 2 BLACS is now built-in with the ScaLAPACK library.

ScaLAPACK

The ScaLAPACK libraries are found on all systems.

Since the usage of ScaLAPACK depends on LAPACK, it involves multiple libraries.

NOTE: As of version 2, ScaLAPACK includes BLACS. This means that it is tightly coupled to the MPI implementation used to build it. The library has therefore changed name to accomodate this.

In order to use this library, first load the compiler and the corresponding MPI libraries you wish to use, as well as scalapack, lapack and blas for that compiler. This example shows the modules you would need to load if you are using OpenMPI, PathScale, and openblas.

module load openmpi/psc
module load scalapack/psc
module load lapack/psc/<version>
module load openblas/psc

Then you link like this (for OpenMPI with LAPACK and OpenBLAS modules loaded):

$SCALAPACK_LDFLAGS -lscalapack_ompi $LAPACK_LDFLAGS $OPENBLAS_LDFLAGS $LAPACK_LIBS $OPENBLAS_LIBS

If it is a C program, you also need to add

-lpathfortran

when using the Pathscale compilers.

FFTW

There are two versions of FFTW available, version 2.1.5 and version 3.x. Both have MPI support. Note that the API has changed between 2.1.5 and the 3.x versions.

Do a module avail and look for libfftw to see which versions are available. To use FFTW, load the module for fftw:

module load fftw/<version>

Then run

module show fftw/<version> 

to see how to use it.

Note that if you need MPI support you must load the MPI libraries first:

module load openmpi/<compiler>
module load fftw/2.1.5

OR

module load openmpi/<compiler>
module load fftw/3.3.3

Then you link with (Example is for a C program, OpenMPI)

mpicc program.c -lfftw3 -lm $FFTW_INCLUDE $FFTW_LDFLAGS

FFT from ACML

You can also use the FFT included with the ACML libraries. First you must load the module:

$ module load acml/<compiler>

To compile and link your program use (remember to also load the compiler you wish to use):

Example, (Pathscale, Fortran program, non-threaded version)

pathf90 program.f90 -lm $ACML_LDFLAGS $ACML_LIBS

Example, (Pathscale, Fortran program, threaded version)

pathf90 program.f90 -lm $ACML_MP_LDFLAGS $ACML_MP_LIBS -lopenmp -pthread 

To see alternative versions, do a module avail and look for acml. For information about usage in your makefile, see module help acml/<compiler> after loading the module.

ParMETIS

ParMETIS is an MPI-based parallel library that implements a variety of algorithms for partitioning unstructured graphs, meshes, and for computing fill-reducing orderings of sparse matrices.

Metis is bundled with Parmetis. First you must load the module:

module load parmetis

To compile and link your program, use (Example is using Pathscale, compiling a Fortran 90 program):

pathf90 program.f -o program -lparmetis $PARMETIS_INCLUDE $PARMETIS_LDFLAG $PARMETIS_LIBS 
Using the bundled Metis: 
pathf90 program.f -o program -lparmetis -lmetis $METIS_INCLUDE $METIS_LDFLAG $METIS_LIBS 

You can also do

module show parmetis

to see how to use it.

There is a manual with more information about usage here [PDF]. There is more information here.

Petsc

Compiled for GCC, OpenMPI. First you need to load the module:

module load petsc

Information about compiling and linking with Petsc, as well as useful examples, can be found here: http://www.mcs.anl.gov/petsc/documentation/

RECSY

RECSY is a library for solving triangular Sylvester-type matrix equations.

Recsy requires a blas implementation to link with like openblas or mkl. The desired blas-module must be loaded as well. In my examples below, I am using openblas.

To use it, load the recsy module (for the desired compiler), as well as either openblas or mkl (here openblas):

module load openblas/<compiler>
module load recsy/<compiler> 

Then link with it (and blas):

$RECSY_LDFLAGS $OPENBLAS_LDFLAGS -lrecsy $OPENBLAS_LIBS 

To read more about RECSY and to see a list of routines, look at the RECSY page.

Slepc

Compiled for GCC, OpenMPI. To use it, you must first load the module:

module load slepc

For information about compiling and linking, as well as useful examples, go here: http://www.grycap.upv.es/slepc/handson/

SLICOT

The subroutine library SLICOT provides Fortran 77 implementations for computations in systems and control theory.

You also need to load a suitable Lapack and Blas to use Slicot.
Example (openblas and lapack)

module load openblas/<compiler>
module load lapack/<compiler>

Do a module avail and look for libslicot to see which versions are available. To use SLICOT, first load the desired compiler, then load the appropriate modules

module load <compiler>
module load libslicot/<compiler>

Link with (also needs blas and lapack, as shown):

$SLICOT_LDFLAGS $LAPACK_LDFLAGS $OPENBLAS_LDFLAGS -lslicot $LAPACK_LIBS $OPENBLAS_LIBS

For more information you can also use

module help libslicot/<compiler>

ACML

ACML (AMD Core Math Library) provides a free set of thoroughly optimized and threaded math routines for HPC, scientific, engineering and related compute-intensive applications.

It contains libraries for:

  • BLAS
  • LAPACK
  • FFT (Fast Fourier Transforms)
  • RNG (Random Number Generators)

In order to load ACML, you must also tell it which compiler you wish to use (psc, intel, pgi, gcc):

$ module load acml/<compiler>

To compile and link your program use (remember to also load the compiler you wish to use):

Example, (Pathscale, Fortran program, non-threaded version)

pathf90 program.f90 $ACML_LDFLAGS $ACML_LIBS

Example, (Pathscale, Fortran program, threaded version)

pathf90 program.f90 $ACML_MP_LDFLAGS $ACML_MP_LIBS -lopenmp -pthread

To see alternative versions, do a module avail and look for acml. For information about usage in your makefile, see

module help acml/<compiler>

after loading the module.

There are more information and examples for linking with ACML in the user guide here on their own website (intel, gcc, pgi examples), but be aware that not all of it pertains to usage on HPC2N.

Intel Math Kernel Library (MKL)

The Intel MKL libraries contains:

  • ScaLAPACK
  • LAPACK
  • Sparse Solver
  • BLAS
  • Sparse BLAS
  • PBLAS
  • GMP
  • FFTs
  • BLACS
  • VSL
  • VML

More information about MKL and the libraries in it can be found here:

Linking with MKL libraries

To use the MKL libraries load the module:

module load mkl

In your Makefile

Add $MKL_LDFLAGS to LDFLAGS
Add $MKL_INCLUDE to your CFLAGS, CXXFLAGS or FFLAGS

Like this:

LDFLAGS=$$MKL_LDFLAGS
CFLAGS=$$MKL_INCLUDE

To correctly use MKL it is vital to have read the documentation. It is available in $base/doc

Commandline:
Here is one examples showing how to link with the MKL libraries. (Line split at \ due to space)

Example, sequential Blas, Intel Fortran

ifort myprogram.f $MKL_LDFLAGS -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -o myprogram

There are too many libraries in MKL to show a complete list of combinations. We refer you to the MKL documentation for examples and support@hpc2n.umu.se for help.

There are two basic version of the library, a non-threaded serial version and a threaded smp version.

Updated: 2017-12-15, 10:08