VASP
Policy
The Vasp program is not distributed via site licences. However, HPC2N have access to the VASP code to be able to support any research groups that has a valid VASP license.
See the VASP license for information regarding terms for published work.
When you have gotten access to a license, the license holder should contact support@hpc2n.umu.se with the following information: license number and list of users who should have access. You will then be given access to using VASP.
Note: only the owner of the license can add/delete users to/from the access list.
General
VASP is a package for performing ab-initio quantum-mechanical molecular dynamics (MD) using pseudopotentials and a plane wave basis set.
Description
VASP is a complex package for performing ab-initio quantum-mechanical molecular dynamics (MD) simulations using pseudopotentials or the projector-augmented wave method and a plane wave basis set. The approach implemented in VASP is based on the (finite-temperature) local-density approximation with the free energy as variational quantity and an exact evaluation of the instantaneous electronic ground state at each MD time step.
VASP uses efficient matrix diagonalisation schemes and an efficient Pulay/Broyden charge density mixing. These techniques avoid all problems possibly occurring in the original Car-Parrinello method, which is based on the simultaneous integration of electronic and ionic equations of motion.
The interaction between ions and electrons is described by ultra-soft Vanderbilt pseudopotentials (US-PP) or by the projector-augmented wave (PAW) method. US-PP (and the PAW method) allow for a considerable reduction of the number of plane-waves per atom for transition metals and first row elements. Forces and the full stress tensor can be calculated with VASP and used to relax atoms into their instantaneous ground-state.
Availability
At HPC2N we have a site installation available on Akka (4.6 and 5.2) and Abisko (5.2) for those with a valid VASP contract/license. We can adapt the installed binaries to suit the groups needs.
Usage at HPC2N
The binaries for vasp are available at /lap/vasp/bin for users authenticated to use them.
Pseudo potentials are installed under /lap/vasp5/potpaw_LDA and /lap/vasp5/potpaw_PBE.
There are several different builds of VASP available, some of them are not available on all clusters yet.
- vasp - built with no special flags
- vasp.noncoll - a softlink to vasp to emphasize that it can be used for noncollinear runs
- vasp.vtst.noncoll - a version with the TRANSITION STATE TOOLS FOR VASP compiled in
- vasp.NGXhalf - built with -DNGXhalf
- vasp.wNGXhalf.NGXhalf - built with -DwNGXhalf -DNGXhalf
- vasp.wNGXhalf - built with -DwNGXhalf
- vasp.mpi - built with no special flags for mpi
- vasp.noncoll.mpi - a softlink to vasp.mpi to emphasize that it can be used for noncollinear runs
- vasp.vtst.noncoll.mpi - a version with the TRANSITION STATE TOOLS FOR VASP for mpi compiled in
- vasp.NGZhalf.mpi - built with -DNGZhalf for mpi
- vasp.wNGZhalf.NGZhalf.mpi - built with -DwNGZhalf -DNGZhalf for mpi
- vasp.wNGZhalf.mpi - built with -DwNGZhalf for mpi
All mpi versions of VASP have been built with scalapack support and with LONGCHAR turned on.
If you need a differently compiled VASP binary, please contact support@hpc2n.umu.se.
Memory consumption guidelines
To be able to run successfully one has to calculate how much memory VASP is going to use and then request the correct amount in the submitfile.
To figure out how much memory your input data will consume you should run the makeparam program, /lap/vasp/bin/makeparam, in the directory where your input data is. There are 3 different instances of the program, makeparam, makeparam.NGXhalf and makeparam.wNGXhalf.NGXhalf. They correspond to the vasp binary with the same name. To get values for the mpi version of vasp use makeparam for vasp.mpi, makeparam.NGXhalf for vasp.NGZhalf.mpi and so on.
Take "gives a total of" value from makeparam, called basemem. Multiply the two largest of NGX/NGY/NGZ, called maxNG. If this is a mpi run then
extramem=((NBANDS+2)*(NRPLWV*2+3)+(maxNG/numcpus))*ISPIN*16/10^6 mpimem=400 (MB)
else
extramem=0 mpimem=0
Then calculate totalmem
totalmem=(basemem/10^6)/numcpus+NPAR*extramem/numcpus+mpimem
All this adds up to the following:
Akka
- set pvmem to 300mb+totalmem
- set pmem to totalmem
- make sure pvmem*x (ppn=x) <~40GB on Akka
- make sure pmem*x (ppn=x) <~15GB on Akka
Exampel using 8cpus with NPAR=4 and ISPIN=2:
akka% makeparam
memory requirements are:
energy cutoff: ENCUT 3400.00 energy cutoff (augmentation): ENAUG 3400.00 memory requirements are: number of kpoints NKPTS 21 number of bands NBANDS 166 number of grid points MPLWV 13436928 number of plane waves NRPLWV 118883 real space IRMAX 1 IRDMAX 431066 arrays on large grid (10 + NTYP)*MPLWVM*16 5159780352 1 sets of wavefunctions 13305811008 projectors in real space 0 projectors in reciprocal space 1479855584 --------------------------------------------------------- gives a total of 19945.4 Mbytes or 19945446944 add 5-10 Mbytes for executable (depends on OS) NGX = 54; NGY = 54; NGZ =576 NGXF=108; NGYF=108; NGZF=*** NBANDS= 166; NBLK= 256
Then
- basemem=19945446944
- maxNG=576*54 (NGZ*NGY)
- extramem=((166+2)*(118883*2+3)+(maxNG/8))*ISPIN*16/10^6
- mpimem=400 (since we are using mpi)
- totalmem=(basemem/10^6)/8+4*extramem/8+mpimem=3600 (rounded upwards)
Since 3600mb*8cpus > 15GB we have to use more then 1 node. This gives us the following submitfile:
#PBS -l nodes=2:ppn=4 #PBS -l pvmem=3900 #PBS -l pmem=3600m
Note: if the calculated values of pvmem and pmem are lower then the system default you should not specify them in the submitfile. I.e. on Akka if pvmem<2000 or pmem<1900 then use the defaults.
Remember that this is only an approximation, but it is as close as we have been able to figure out. There are also situations where this still isn't enough to avoid running out of memory. If this happens then increase pvmem only in increments of 1000m while still remaining below the limit for the system.
Abisko
- You should normally not have to set --mem
- The default memory if ~2500MB/core, i.e. <# cores>*2500MB
- make sure mem*x (cores=x) <~128GB on Abisko, unless running on the 10 nodes that have 512GB memory.
Otherwise, the memory can be found in the same way as for Akka.
If any of this is unclear please send a mail to support@hpc2n.umu.se and we will try to make it more understandable.
Submit file examples
Akka, Serial version
#!/bin/bash #PBS -A Your-SNIC-account #PBS -N vasp-serial-test #PBS -l nodes=1:ppn=1 #PBS -l walltime=00:30:00 #PBS -l pvmem=8000mb #PBE -l pmem=6800mb cd $PBS_O_WORKDIR module add vasp vasp
Akka, MPI version
#!/bin/bash #PBS -A Your-SNIC-account #PBS -N vasp-mpi-test #PBS -l nodes=4:ppn=2 #PBS -l walltime=00:30:00 #PBS -l pvmem=4000mb #PBS -l pmem=3400mb cd $PBS_O_WORKDIR module add vasp mpiexec vasp.NGZhalf.mpi
Abisko, Serial version
#!/bin/bash #SBATCH -o vasp.%j.out #SBATCH -J my_vasp_job #SBATCH -A SNICXXX-YY-ZZ #SBATCH -n 1 #SBATCH --time=10:00:00 # Load modules, unless already done before job is submitted (remember, # SLURM export the environment as per default, unless you # set --export=NONE module load vasp vasp
Abisko, MPI version (three different examples)
#!/bin/bash #SBATCH -o vasp.%j.out #SBATCH -J my_vasp_job #SBATCH -A SNICXXX-YY-ZZ #SBATCH -N 1 #SBATCH -n 12 #SBATCH --time=10:00:00 # Load modules, unless already done before job is submitted (remember, # SLURM export the environment as per default, unless you # set --export=NONE module load vasp srun vasp.mpi
#!/bin/bash #SBATCH -A SNICXXX-YY-ZZ #SBATCH -J my_vasp_job #SBATCH -n 24 #SBATCH --time=04:00:00 #SBATCH --ntasks-per-node=12 module load vasp srun vasp.NGZhalf.mpi
#!/bin/bash #SBATCH -A SNICXXX-YY-ZZ #SBATCH -J my_vasp_job #SBATCH -N 4 #SBATCH --exclusive #SBATCH -t 20:00:00 module load vasp srun -n 96 --ntasks-per-socket=6 vasp.NGZhalf.mpi > out 2>&1 & cd <path_to_other_directory> srun -n 96 --ntasks-per-socket=6 vasp.NGZhalf.mpi > out 2>&1 wait
Additional info
Documentation is available on the VASP homepage.
