OpenFOAM

Policy

OpenFOAM is produced by OpenCFD Ltd, is freely available and open source, licensed under the GNU General Public Licence.

General

The OpenFOAM (Open Field Operation and Manipulation) CFD Toolbox can simulate anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics, electromagnetics and the pricing of financial options.

Description

The core technology of OpenFOAM is a flexible set of efficient C++ modules. These are used to build a wealth of: solvers, to simulate specific problems in engineering mechanics; utilities, to perform pre- and post-processing tasks ranging from simple data manipulations to visualisation and mesh processing; libraries, to create toolboxes that are accessible to the solvers/utilities, such as libraries of physical models.

OpenFOAM is supplied with numerous pre-configured solvers, utilities and libraries and so can be used like any typical simulation package. However, it is open, not only in terms of source code, but also in its structure and hierarchical design, so that its solvers, utilities and libraries are fully extensible.

OpenFOAM uses finite volume numerics to solve systems of partial differential equations ascribed on any 3D unstructured mesh of polyhedral cells. The fluid flow solvers are developed within a robust, implicit, pressure-velocity, iterative solution framework, although alternative techniques are applied to other continuum mechanics solvers. Domain decomposition parallelism is fundamental to the design of OpenFOAM and integrated at a low level so that solvers can generally be developed without the need for any parallel-specific coding.

Availability

On HPC2N we have OpenFOAM available as a module on Akka and Abisko.

Usage at HPC2N

To use, add the openfoam module to add it to your environment. There may be multiple versions installed, use module avail openfoam to show the available versions and specify the version you want to use in the module add command. This shows how to add the module for OpenFOAM 2.0

module add openfoam/2.0

Notes

• HPC2N has compiled with Intel compilers and there are some patches applied, which means that some third party applications/modules may not compile out of the box.
• OpenFOAM provides a few handy environment variables that refers to various directories, please use them and not hard coded pathnames as this will make your life easier when OpenFOAM gets upgraded.
• the OpenFOAM documentation refers to editing shell settings such as .bashrc and .cshrc, this is not needed and might cause unexpected behaviour since module takes care of setting all needed variables.

Serial jobs

There are some examples in the tutorials directory of the OpenFOAM installation ($FOAM_TUTORIALS). Here we will look at $FOAM_TUTORIALS/incompressible/icoFoam/cavity and we will run OpenFOAM 2.0.

In order to run a serial job like the one in the directory above, you should do the following:

1. module add openfoam/2.0

2. Create your working directory (won't delete existing). This should be done in pfs, if you want to run batch jobs:

   mkdir -p $FOAM_RUN

3. Change to that directory:

   cd $FOAM_RUN

4. Copy the tutorials there and change permissions:

   cp -r $FOAM_TUTORIALS tutorials
   chmod -R 755 tutorials
   

5. The following is correct for OpenFOAM 1.6 and higher. For 1.5, the directory is tutorials/icoFoam/cavity

   cd tutorials/incompressible/icoFoam/cavity

6. There will always be (at least) three subdirectories. Cases for OpenFOAM are setup by editing case files. A case being simulated involves data for mesh, fields, properties, control parameters, etc. The structure can be seen here: File Structure of OpenFOAM cases. The three subdirectories that are always present are:
   1. 0: containing the files p and U, with information about the boundary and initial conditions for the pressure and the velocity. More information can be found in the example here: Lid-driven cavity flow. There can be more than one 'time' directory.
   2. constant: containing the directory polymesh and one or more files with the suffix ...Properties. polymesh has files for mesh generation, and the ...Properties are files for the physical properties. In the case of icoFoam, the only property that needs to be specified is the kinematic viscosity (in transportProperties)
   3. system: this directory contains files for controlling the case (controlDict), discretisation schemes (fvSchemes), the specification of linear solvers and tolerances (fvSolution), and other things like setting the initial field (setFields), depending on the case. They can be found in the OpenFOAM User Guide.
7. In order to run the case, you must either be located in the directory or give the path to it.
8. The first you must do is run blockMesh to generate the mesh - several needed files.

   $ blockMesh
   

9. It is sometimes a good idea to view the mesh to check for any errors before running. You can read more about how to do this here: Viewing the mesh.
10. You then run the application by typing the name of the solver or utility (here the solver icoFoam) while standing in the case directory, or with the path (here icoFoam -case $path_to/tutorials/incompressible/icoFoam/cavity)

    $ icoFoam
    

Parallel jobs

The example we will look at is tutorials/multiphase/interFoam/laminar/damBreak. The example will be for OpenFOAM 2.0, but it should not be different for earlier versions (1.6 is tested to work in the same manner). Remember, if you want to submit OpenFOAM batch jobs, your working directory has to be on pfs.

You probably need to make some changes to the example.

1. module load openfoam/2.0

2. First, make a copy to make the changes in:

   cd $FOAM_RUN/tutorials/multiphase/interFoam/laminar
   

3. Create a new directory to play in

   mkdir damBreakFine
   

4. Copy all the files to it

   cp -r damBreak/0 damBreakFine
   cp -r damBreak/system damBreakFine
   cp -r damBreak/constant damBreakFine
   

5. Enter the new case directory and change the blocks description in the blockMeshDict dictionary (in the directory /constant/polyMesh) to

   blocks
   (	    
       hex (0 1 5 4 12 13 17 16) (46 10 1) simpleGrading (1 1 1)
       hex (2 3 7 6 14 15 19 18) (40 10 1) simpleGrading (1 1 1)
       hex (4 5 9 8 16 17 21 20) (46 76 1) simpleGrading (1 2 1)
       hex (5 6 10 9 17 18 22 21) (4 76 1) simpleGrading (1 2 1)
       hex (6 7 11 10 18 19 23 22) (40 76 1) simpleGrading (1 2 1)
   );
   

6. Run blockMesh (in the damBreakFine/ directory) to create the mesh etc.

   $ blockMesh
   

7. As the mesh has now changed from the damBreak example, the user must re-initialise the phase field alpha1 in the 0 time directory since it contains a number of elements that is inconsistent with the new mesh. The best way to do this, is to rerun the setFields utility. There is a backup copy of the initial uniform α1 that the user should copy to 0/alpha1 before running setFields:

   cd $FOAM_RUN/tutorials/multiphase/interFoam/laminar/damBreakFine
   cp -r 0/alpha1.org 0/alpha1
   setFields
   

8. The method of parallel computing used by OpenFOAM is known as domain decomposition, in which the geometry and associated fields are broken into pieces and allocated to separate processors for solution. The first step required to run a parallel case is therefore to decompose the domain using the decomposePar utility. There is a dictionary associated with decomposePar named decomposeParDict which is located in the system directory of the tutorial case:

   cd $FOAM_RUN/tutorials/multiphase/interFoam/laminar/damBreakFine/system
   

9. Open decomposeParDict in your favourite editor. The first entry is numberOfSubdomains which specifies the number of subdomains into which the case will be decomposed, usually corresponding to the number of processors available for the case.
10. In this example we are using 8 processors, so:

    numberOfSubdomains 8;
    

    There is information in section 3.4 of the User Guide with further details of how to run a case in parallel.

11. After this parameter is sat, you should run (in damBreakFine)

    $ decomposePar
    

    in order to automatically construct subdirectories. These will have been created, one for each processor, in the case directory. The directories are named processorN, where N = 0,1, ...

12. To run this, you need to submit a batch job. How to do this depends on whether you run on Akka or on Abisko.
    • Akka: Make a job submit file containing at least this (remember, you must submit from the case directory OR change $PBS_O_WORKDIR to the path to the case directory. This example used 1 node and 8 processors).

      #!/bin/bash
      # Change the project below to match your actual project
      #PBS -A SNICXXX-YY-ZZ
      # name of the output file (not strictly necessary, but a good idea)
      #PBS -o openfoam_dambreak.out
      # name of the error file (not strictly necessary, but a good idea)
      #PBS -e openfoam_dambreak.err
      
      # asking for 1 node on Akka
      #PBS -l nodes=1
      # the job is allowed to use up to 30 minutes to run
      #PBS -l walltime=00:30:00
      
      cd $PBS_O_WORKDIR
      
      module load openfoam/2.0
      
      mpiexec interFoam -parallel
      

      Submit the job from the case directory (or give the path in the submit file) with

      $ qsub <job_script> 

    • Abisko: Make a job submit file like this (works for 1 node and 16 processors - just change 8 to 16 in the example above). Remember, SLURM exports the environment (including modules), so if you have already loaded the module files for OpenFOAM, you should not do so again in the script.

      #!/bin/bash
      #SBATCH -A staff
      # Asking for 1 node and 16 processors on Abisko
      #SBATCH -N 1 
      #SBATCH -n 16
      #SBATCH --time=00:15:00
      
      module load openfoam/2.0
      
      srun --resv-ports mpiexec interFoam -parallel
      

      Submit the job from the case directory (or give the path in the submit file) with

      sbatch <job submit file>

13. The above example works on a local multiprocessor machine (like Akka or Abisko). It is possible to run across a network. Look at section 3.4.2 in the User Guide for information about that.

Extending OpenFOAM

OpenFOAM is often used as a framework when developing own codes for solving problems. Applications come in two main categories; solvers and utilities. It is often possible to find an already existing application that is similar to what you would like to do. Copy that and modify it for your purposes. When reading the OpenFOAM documentation on the issue, please note the following regarding where to place your binaries/libraries in order for it to work:

$WM_PROJECT_USER_DIR is the OpenFOAM user project directory which contains $FOAM_USER_APPBIN and $FOAM_USER_LIBBIN. The name of these directories are configurable, so don't trust the documentation which uses absolute paths in some places. Use the variables instead.

The root path for the OpenFOAM user project directory is $WM_PROJECT_USER_DIR (type env if you need to see the actual path). If you have questions about the different options available, please contact support@hpc2n.umu.se.

Unfortunately, the documentation on the issue of variables/paths is a bit unclear and contradicting so we'll try to clarify a few things:

• All environment variables needed are set by the module system, so you should not set them in bashrc/cshrc. They will be overridden when you load the module.
• If you compile your own binary, place it in $FOAM_USER_APPBIN so it will be found before the OpenFOAM binaries. $FOAM_USER_APPBIN is included in the search path for binaries ($PATH).
• Use the environment variables ($FOAM_USER_APPBIN and $FOAM_USER_LIBBIN) to specify destination directories, this ensures that correct directories are specified and eases future upgrades. For example, if you're building an application called bottledbanana, your Make/files file should contain:

  EXE = $(FOAM_USER_APPBIN)/bottledbanana
  

The general steps of compiling your own application is (the below case is for a solver - just change to utility if that is what you want):

• The applications can be found in $WM_PROJECT_DIR/applications
• Copy an application that is similar to what you want to achieve (or use one you have written yourself or downloaded)
  • If you want to modify an existing application, change to the $WM_PROJECT_DIR and follow these steps:
  • cp -riuv --parents --backup applications/solvers/<path to solver> $WM_PROJECT_USER_DIR
  • cd $WM_PROJECT_USER_DIR/applications/solvers/<solver>
  • rename solver directory to <Your_solver_directory>
  • cd <Your_solver_directory>
  • wclean (just to be safe)
  • rename and modify necessary files, including the Makefile (in Make/files)
  • run wmake in Your_solver_directory. Be prepared for a lot of warnings. They should be inconsequential.
  • If needed, modify cases as necessary

For further help, here is an example of compiling your own OpenFOAM application.

Additional info

• OpenFOAM documentation: User Guide.
• Information about running applications in parallel.
• General information about running applications.
• Compilation of your own applications and libaries are covered in 'Compiling applications and libraries'.
• There exists an 'Unofficial OpenFOAM wiki'.
• Additional documentation and examples can be found at PhD course in CFD with OpenSource software, 2009 (Chalmers)