The OpenFOAM (Open Field Operation and Manipulation) CFD Toolbox can simulate anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics, electromagnetics and the pricing of financial options.
The core technology of OpenFOAM is a flexible set of efficient C++ modules. These are used to build a wealth of: solvers, to simulate specific problems in engineering mechanics; utilities, to perform pre- and post-processing tasks ranging from simple data manipulations to visualisation and mesh processing; libraries, to create toolboxes that are accessible to the solvers/utilities, such as libraries of physical models.
OpenFOAM is supplied with numerous pre-configured solvers, utilities and libraries and so can be used like any typical simulation package. However, it is open, not only in terms of source code, but also in its structure and hierarchical design, so that its solvers, utilities and libraries are fully extensible.
OpenFOAM uses finite volume numerics to solve systems of partial differential equations ascribed on any 3D unstructured mesh of polyhedral cells. The fluid flow solvers are developed within a robust, implicit, pressure-velocity, iterative solution framework, although alternative techniques are applied to other continuum mechanics solvers. Domain decomposition parallelism is fundamental to the design of OpenFOAM and integrated at a low level so that solvers can generally be developed without the need for any parallel-specific coding.
On HPC2N we have OpenFOAM and OpenFOAM-Extend available as a modules.
To use, add the openfoam or openfoam-extend module to add it to your environment. There are multiple versions installed, so you need to use:
ml spider openfoam # or openfoam-extend
to see which versions are available and then how to load the module and the needed prerequisites:
ml spider OpenFOAM/<version>
Example, loading openfoam version 4.1
ml icc/2017.1.132-GCC-6.3.0-2.27 ml impi/2017.1.132 ml OpenFOAM/4.1
After loading the module, you also need to do:
- You have to make sure that $HOME/OpenFOAM and $HOME/foam (OpenFOAM-extend uses $HOME/foam) are softlinks pointing to either $HOME/Public/OpenFOAM (and foam respectively) or into your /pfs/nobackup directory. If you do not know what this means, please ask firstname.lastname@example.org
- FOAM_RUN is set to $WM_PROJECT_USER_DIR/run
- It is best to make sure that the $FOAM_RUN directory is located under your /pfs/nobackup tree, using softlinks, to be able to run batch jobs.
- HPC2N has compiled OpenFOAM(-Extend) with the Intel compilers and there are some patches applied, which means that some third party applications/modules may not compile out of the box.
- OpenFOAM(-Extend) provides a few handy environment variables that refers to various directories, please use them and not hard coded pathnames as this will make your life easier when OpenFOAM gets upgraded. You can find them with "env|grep FOAM"
- the OpenFOAM(-Extend) documentation refers to editing shell settings such as .bashrc and .cshrc, this is not needed and might cause unexpected behaviour since loading the module and sourcing $FOAM_BASH takes care of setting all needed variables.
- Important! You should NOT attempt to use the cases and applications in the tutorials for a different version of OpenFOAM! Some of them may work, but most will not. Make a fresh copy of the applications and cases you wish to modify, from the version of OpenFOAM you wish to run.
- paraFoam does not work on HPC2N, so that part will have to be done on your own computer.
- Use the environment variables ($FOAM_USER_APPBIN and $FOAM_USER_LIBBIN) to specify destination directories when building your own extensions, this ensures that correct directories are specified and eases future upgrades.
- $FOAM_USER_APPBIN is included in the search path for binaries ($PATH).
There are some examples in the tutorials directory of the OpenFOAM installation ($FOAM_TUTORIALS). Here we will look at $FOAM_TUTORIALS/incompressible/icoFoam/cavity and we will run OpenFOAM 4.1.
In order to run a serial job like the one in the directory above, you should do the following:
ml icc/2017.1.132-GCC-6.3.0-2.27 impi/2017.1.132 OpenFOAM/4.1
- Create your working directory (won't delete existing). This should be done in pfs, if you want to run batch jobs:
mkdir -p $FOAM_RUN
- Change to that directory:
- Copy the tutorials there and change permissions:
cp -r $FOAM_TUTORIALS tutorials chmod -R 755 tutorials
- The following is correct for OpenFOAM 4.0 and higher. For 1.5, the directory is tutorials/icoFoam/cavity and for 2.x and 3.x it is tutorials/incompressible/icoFoam/cavity
- There will always be (at least) three subdirectories. Cases for OpenFOAM are setup by editing case files. A case being simulated involves data for mesh, fields, properties, control parameters, etc. The structure can be seen here: File Structure of OpenFOAM cases. The three subdirectories that are always present are:
- 0: containing the files p and U, with information about the boundary and initial conditions for the pressure and the velocity. More information can be found in the example here: Lid-driven cavity flow. There can be more than one 'time' directory.
- constant: containing the directory polymesh and one or more files with the suffix ...Properties. polymesh has files for mesh generation, and the ...Properties are files for the physical properties. In the case of icoFoam, the only property that needs to be specified is the kinematic viscosity (in transportProperties)
- system: this directory contains files for controlling the case (controlDict), discretisation schemes (fvSchemes), the specification of linear solvers and tolerances (fvSolution), and other things like setting the initial field (setFields), depending on the case. They can be found in the OpenFOAM User Guide.
- In order to run the case, you must either be located in the directory or give the path to it.
- The first you must do is run blockMesh to generate the mesh - several needed files.
- It is sometimes a good idea to view the mesh to check for any errors before running. You can read more about how to do this here: Viewing the mesh (go down to section 1.1.2).
- You then run the application by typing the name of the solver or utility (here the solver icoFoam) while standing in the case directory, or with the path (here icoFoam -case $path_to/tutorials/incompressible/icoFoam/cavity/cavity)
The example we will look at is tutorials/multiphase/interFoam/laminar/damBreak. The example will be for OpenFOAM 4.1, but it should not be different for earlier versions (1.6 is tested to work in the same manner). Remember, if you want to submit OpenFOAM batch jobs, your working directory has to be on pfs.
You probably need to make some changes to the example. Note that this example assumes you copied the tutorials as shown under the serial example.
ml icc/2017.1.132-GCC-6.3.0-2.27 impi/2017.1.132 OpenFOAM/4.1
- First, make a copy to make the changes in:
- Create a new directory to play in
- Copy all the files to it (earlier versions have the files directly under the first "damBreak")
cp -r damBreak/damBreak/0 damBreakFine cp -r damBreak/damBreak/system damBreakFine cp -r damBreak/damBreak/constant damBreakFine
- Enter the new case directory and change the blocks description in the blockMeshDict dictionary (in the directory /system) to
blocks ( hex (0 1 5 4 12 13 17 16) (46 10 1) simpleGrading (1 1 1) hex (2 3 7 6 14 15 19 18) (40 10 1) simpleGrading (1 1 1) hex (4 5 9 8 16 17 21 20) (46 76 1) simpleGrading (1 2 1) hex (5 6 10 9 17 18 22 21) (4 76 1) simpleGrading (1 2 1) hex (6 7 11 10 18 19 23 22) (40 76 1) simpleGrading (1 2 1) );
- Run blockMesh (in the damBreakFine/ directory) to create the mesh etc.
- As the mesh has now changed from the damBreak example, the user must re-initialise the phase field alpha1 in the 0 time directory since it contains a number of elements that is inconsistent with the new mesh. The best way to do this, is to rerun the setFields utility. There is a backup copy of the initial uniform α1 that the user should copy to 0/alpha1 before running setFields:
cd $FOAM_RUN/tutorials/multiphase/interFoam/laminar/damBreakFine cp -r 0/alpha.water.orig 0/alpha.water setFields
- The method of parallel computing used by OpenFOAM is known as domain decomposition, in which the geometry and associated fields are broken into pieces and allocated to separate processors for solution. The first step required to run a parallel case is therefore to decompose the domain using the decomposePar utility. There is a dictionary associated with decomposePar named decomposeParDict which is located in the system directory of the tutorial case:
- Open decomposeParDict in your favourite editor. The first entry is numberOfSubdomains which specifies the number of subdomains into which the case will be decomposed, usually corresponding to the number of processors available for the case.
- In this example we are using 16 processors, so:
We also need to adjust n = nxnynz in simpleCoeffs accordingly, so nxny = numberOfSubdomains. There is information in section 3.1 of the User Guide (scroll down to 3.1.11) with further details of how to run a case in parallel.
- After this parameter is sat, you should run (in damBreakFine)
in order to automatically construct subdirectories. These will have been created, one for each processor, in the case directory. The directories are named processorN, where N = 0,1, ...
- To run this, you need to submit a batch job. Depending on which cluster you run on, you can have different amounts of cores. Abisko has 48 cores per node, while Kebnekaise has 28 cores per node (for the regular compute nodes - see the Kebnekaise hardware page for information about the other node types).
Make a job submit file like this (works for 16 cores). Remember, SLURM exports the environment (including modules), so you should do ml purge first to make sure that the submit file loads the expected module.
#!/bin/bash #SBATCH -A SNICXXXX-YY-ZZ #SBATCH --output=openfoam_dambreak.out #SBATCH --error=openfoam_dambreak.err # Asking for 16 cores #SBATCH -n 16 #SBATCH --time=00:15:00 #For OpenFOAM version 4.1 ml purge > /dev/null 2>&1 # Ignore some warnings from the purge command ml icc/2017.1.132-GCC-6.3.0-2.27 impi/2017.1.132 OpenFOAM/4.1 source $FOAM_BASH srun interFoam -parallel
Submit the job from the case directory (or give the path in the submit file) with
sbatch <job submit file>
The above example works on a local multiprocessor machine (like Abisko). It is possible to run across a network. Look at section 3.4.2 in the User Guide for information about that.
Running on all 48 cores of 1 node (Abisko).
#!/bin/bash # Put in actual SNIC number #SBATCH -A SNICXXXX-YY-ZZ # Asking for 1 node (the whole node) on Abisko #SBATCH -n 48 #SBATCH --time=00:15:00 #For OpenFOAM version 4.1 ml purge > /dev/null 2>&1 # Ignore warnings from purge ml icc/2017.1.132-GCC-6.3.0-2.27 impi/2017.1.132 OpenFOAM/4.1 source $FOAM_BASH srun -cpu_bind=cores interFoam -parallel
Running on 32 cores on 1 node (Abisko) using all the memory.
#!/bin/bash # Put in actual SNIC number #SBATCH -A SNICXXXX-YY-ZZ # Asking for 1 node (the whole node) on Abisko #SBATCH -n 32 #SBATCH --exclusive #SBATCH --time=00:15:00 #For OpenFOAM version 4.1 ml purge > /dev/null 2>&1 # Ignore warnings from purge ml icc/2017.1.132-GCC-6.3.0-2.27 impi/2017.1.132 OpenFOAM/4.1 source $FOAM_BASH srun interFoam -parallel
If a job runs slowly due to memory bandwidth limitation, it can be a good idea to try and split over more nodes. In this example, over two nodes (Abisko).
#!/bin/bash # Put in actual SNIC number #SBATCH -A SNICXXXX-YY-ZZ #SBATCH -n 8 #SBATCH -c 6 # use all 6 cores in a NUMA island per MPI task for increased memory bandwidth #SBATCH --time=00:15:00 #For OpenFOAM version 4.1 ml purge > /dev/null 2>&1 # Ignore warnings from purge ml icc/2017.1.132-GCC-6.3.0-2.27 impi/2017.1.132 OpenFOAM/4.1 source $FOAM_BASH srun interFoam -parallel
OpenFOAM is often used as a framework when developing own codes for solving problems. Applications come in two main categories; solvers and utilities. It is often possible to find an already existing application that is similar to what you would like to do. Copy that and modify it for your purposes. When reading the OpenFOAM documentation on the issue, please note the following regarding where to place your binaries/libraries in order for it to work:
$WM_PROJECT_USER_DIR is the OpenFOAM user project directory which contains $FOAM_USER_APPBIN and $FOAM_USER_LIBBIN. The name of these directories are configurable, so don't trust the documentation which uses absolute paths in some places. Use the variables instead.
The root path for the OpenFOAM user project directory is $WM_PROJECT_USER_DIR. If you have questions about the different options available, please contact email@example.com.
For further help, here is an example of compiling your own OpenFOAM application.
- OpenFOAM documentation: User Guide.
- Information about running applications in parallel.
- General information about running applications.
- Compilation of your own applications and libaries are covered in 'Compiling applications and libraries'.
- There exists an 'Unofficial OpenFOAM wiki'.
- Additional documentation and examples can be found at PhD course in CFD with OpenSource software, 2009 (Chalmers)