Installing software with EasyBuild as a user

  • Posted on: 14 March 2017
  • By: bbrydsoe

Installing software modules with EasyBuild as a user, on Kebnekaise

NOTE: If you have read the official EasyBuild instructions there will have been mention of a configuration file. When installing software as a user on Kebnekaise, you do NOT need an EasyBuild configuration file, and it can even cause problems to have one.

This guide will let you install software (and create a module for easy loading) to your $HOME/Public directory, under a subdirectory called easybuild.

After you have installed software with EasyBuild on Kebnekaise, your $HOME/Public/easybuild will contain:

  • ebfiles_repo - easybuild configuration files for software you have already built
  • modules - the modules for the software you have built
  • software - the software you have built 

Installing software

  1. Load the SoftInst module
    ml SoftInst
  2. You can check your EasyBuild config settings with:
    eb --show-config 
    eb --show-full-config 
    
  3. Check if the software you want exists as EasyBuild software package. This can be done in two ways:
    1. Go to the EasyBuild software list
    2. Run the command
      eb -S '^software'

      where software is the name of the software you are looking for.

  4. Pick a toolchain to build with. You should always choose one that is installed on Kebnekaise to build with, and not build a new one. If the software you want to install does not have an EasyBuild configuration for one of our exisiting toolchains, then try to build with one of them anyway, using the flag --try-toolchain=<toolchain> ,<version>. The newest of the installed versions is recommended. You can get a list of toolchains that the software in question has been successfully built with using either of the methods mentioned under 3. Look at
    ml av

    to see which toolchains are available on Kebnekaise.

  5. Do a "dry-run" first to see if it will build or if there are errors:
    eb <software-easybuild-config.eb> --dry-run

    where <software-easybuild-config.eb> is the easybuild configure file you get from either the software list page or from running

    eb -S '^software'
  6. The format of the name of an EasyBuild configure file is <name-of-software>-<version>-<toolchain>-<version-of-toolchain>.eb.
    1. Example: FFmpeg-3.1.3-intel-2016b.eb is for the software FFmpeg, version 3.1.3. The configuration file is for the toolchain intel, version 2016b.
  7. If all goes well, install the software. Note that you should not give the full path to the software's easybuild configure file, only its name:
    eb <software-easybuild-config.eb>
    • In order to try with a different toolchain (e.g. if Kebnekaise does not have the one installed that the software defaults to), then do:
      eb  <software-easybuild-config.eb> --try-toolchain=<toolchain>,<version>
    • The software may have a number of prerequisites which needs to be installed as well. In order to do this automatically, do
      eb  <software-easybuild-config.eb> --robot

      or

      eb  <software-easybuild-config.eb> --try-toolchain=<toolchain>,<version> --robot

      to use a different toolchain than the default for the software.

  8. If you get any errors, contact support@hpc2n.umu.se.
  9. It will take some time to install the software, any prerequisites, and to create the module.
  10. If all went well, you will now have a software module in $HOME/Public/easybuild/modules which can be loaded, just like the system-wide installed modules on Kebnekaise. NOTE: You need to load the toolchain it was built with first.

Example, building FFmpeg with foss/2017a

  1. Pick a toolchain from those offered on the above page with listed software. Or check which configurations exist for a specific software by using
    eb -S '^ffmpeg'

    Here we will do the latter:

    b-an01 [~/pfs]$ eb -S '^ffmpeg'
    CFGS1=/hpc2n/eb/software/Core/EasyBuild/3.1.1/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.1-py2.7.egg/easybuild/easyconfigs/f/FFmpeg
     * $CFGS1/FFmpeg-0.10.16-gimkl-2.11.5.eb
     * $CFGS1/FFmpeg-0.10.16-intel-2016a.eb
     * $CFGS1/FFmpeg-2.4-intel-2014.06.eb
     * $CFGS1/FFmpeg-2.4-intel-2014b.eb
     * $CFGS1/FFmpeg-2.4-intel-2015a.eb
     * $CFGS1/FFmpeg-2.8-intel-2015b.eb
     * $CFGS1/FFmpeg-2.8.4-foss-2015a.eb
     * $CFGS1/FFmpeg-2.8.5-foss-2015a.eb
     * $CFGS1/FFmpeg-2.8.6-intel-2016a.eb
     * $CFGS1/FFmpeg-2.8.7-foss-2016a.eb
     * $CFGS1/FFmpeg-2.8.7-intel-2016a.eb
     * $CFGS1/FFmpeg-3.0.2-foss-2016a.eb
     * $CFGS1/FFmpeg-3.0.2-intel-2016a.eb
     * $CFGS1/FFmpeg-3.1.3-foss-2016b.eb
     * $CFGS1/FFmpeg-3.1.3-intel-2016b.eb
  2. We have foss/2016b, but we are going to try with foss/2017a. In any case, let's pick FFmpeg-3.1.3-foss-2016b.eb
  3. Let us take a look inside the configuration file (FFmpeg-3.1.3-foss-2016b.eb). The path is found from eb -S '^ffmpeg' and looking inside the file gives:
    b-an01 [~/pfs]$ less /hpc2n/eb/software/Core/EasyBuild/3.1.1/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.1-py2.7.egg/easybuild/easyconfigs/f/FFmpeg/FFmpeg-3.1.3-intel-2016b.eb
    
    
    easyblock = 'ConfigureMake'
    
    name = 'FFmpeg'
    version = '3.1.3'
    
    homepage = 'https://www.ffmpeg.org/'
    description = """A complete, cross-platform solution to record, convert and stream audio and video."""
    
    toolchain = {'name': 'foss', 'version': '2016b'}
    
    sources = [SOURCELOWER_TAR_BZ2]
    source_urls = ['http://ffmpeg.org/releases/']
    
    dependencies = [
        ('NASM', '2.12.02'),
        ('zlib', '1.2.8'),
        ('bzip2', '1.0.6'),
        ('x264', '20160614'),
        ('X11', '20160819'),
    ]
    
    configopts = '--enable-pic --enable-shared --enable-gpl --enable-version3 --enable-nonfree --cc="$CC" --cxx="$CXX" '
    configopts += '--enable-libx264'
    
    sanity_check_paths = {
        'files': ['bin/ff%s' % x for x in ['mpeg', 'probe', 'server']] +
                 ['lib/lib%s.%s' % (x, y) for x in ['avdevice', 'avfilter', 'avformat', 'avcodec', 'postproc',
                                                    'swresample', 'swscale', 'avutil'] for y in [SHLIB_EXT, 'a']],
        'dirs': ['include']
    }
    
    moduleclass = 'vis'
  4. We're building with foss/2017a. Let us do a "dry-run" first to see if it will build or if there are errors:
    eb FFmpeg-3.1.3-foss-2016b.eb --try-toolchain=foss,2017a --dry-run

    You can also write this as

    eb FFmpeg-3.1.3-foss-2016b.eb --try-toolchain-version=2017a --dry-run

    if you are only changing the version of the toolchain and not the toolchain itself.

  5. No errors! Let's build!
  6. We are building including any prerequisites:
    eb FFmpeg-3.1.3-foss-2016b.eb --try-toolchain-version=2017a --robot
  7. When the software is installed and the module created, you can load it with:
    ml foss/2017a
    ml FFmpeg/3.1.3
    

References

Updated: 2017-03-22, 15:55