Howto create your own/group software modules

Introduction

Often a program requires particular environment variables in order to run. LMOD environment modules provide a convenient way to meet these requirements in a dynamical way. By creating modulefiles for your softwares it will be easy to add and remove directories to the PATH,LIBRARY_PATH,MANPATH,CLASSPATH and many other environment variables just by loading and unloading the module(file). This will allow for easily changing the environment (variables) and or software versions. Here an introduction on how to create these software module files.

We also have a number of system-provided modules. Before creating your own module, please check whether this is already provided system-wide. For details, see FirstTimeUsers#Environment_software_modules

Case

Lets assume we want program PROGRAM with version VERSION to become available as a software module.

• PROGRAM (e.g. samtools, plink, bedtools)
• VERSION (e.g. 1.3, 2, 2.4.8)

Get and install the program in a standardized way

#$wget https://url/to/PROGRAM-VERSION.tar.gz/download
#$tar -zxvf PROGRAM-VERSION.tar.gz
#$cd PROGRAM-VERSION
#$configure --prefix=$HOME/$MY_DISTRO/software/PROGRAM-VERSION
#$make -j4
#$make install

Make your own (cluster distribution specific) software modules visible when you login

This needs to be added once to your $HOME/.bash_profile After changing your .bash_profile also update your environment by invoking `source ~/.bash_profile`

#MY PRIVATE SOFTWARE MODULES
privatemodules=$HOME/$MY_DISTRO/etc/modulefiles
if [ -d \$privatemodules ];then
   MODULEPATH=$privatemodules:$MODULEPATH
fi

Create a module file for the program

These module files should end up in a path that is defined in the $MODULEPATH environment variable It might be convenient to to name the module after the program name

#$mkdir -p $HOME/$MY_DISTRO/etc/modulefiles/PROGRAM

The lines cat..EOF should be copied in a (script) file and modified according to your (PROGRAMs) needs At least replace PROGRAM and VERSION and execute the file (script)

cat <<EOF>>$HOME/$MY_DISTRO/etc/modulefiles/PROGRAM/VERSION.lua
help(
[[ PROGRAM(version VERSION) Just Another Algorithm. It is a program for analysis of again something using a simulation and not wholly unlike another program.
JAA is licensed under the ShareYourCode License. You may freely modify and redistribute it under certain conditions (see the file COPYING for details). - Homepage: http://JAA.sources.notyet
]])

whatis("JAA (version VERSION) Just Another Algorithm. It is a program for analysis of again something using a simulation not wholly unlike another program. - Homepage: http://JAA.sources.notyet")

local version = "VERSION"

local base = "/hpc/local/\$MY_DISTRO/\$MY_GROUP/software/PROGRAM-" .. version

conflict("PROGRAM")

prepend_path("LD_LIBRARY_PATH", pathJoin(base, "lib"))
prepend_path("LIBRARY_PATH", pathJoin(base, "lib"))
prepend_path("PKG_CONFIG_PATH", pathJoin(base, "lib/pkgconfig"))
prepend_path("MANPATH", pathJoin(base, "share/man"))
prepend_path("PATH", pathJoin(base, "bin"))

EOF

remarks

• In lua lines starting with -- are interpreted as comments
• The number of environment variables that need to be set (prepend_path lines) to make your program running depends on the needs of your program.
• If the program of requires another software module (e.g. a particular Java) to be loaded, this can be automated by adding lines like lines like:

load("Java/1.8.0_60")
prereq("Java/1.8.0_60")

Check module file

Once you have installed the program and created its module file lets see if the module is available:

#$module spider

If it does not appear check if the module (lua) file is in the right path.

#$module load PROGRAM

Should load dependencies (if exists) and report on any changes in the environment In case of (lua) syntax errors it will show the line number that has an error.

Now check if the environment variables have been changed according to the module file and the programs needs e.g.

#$echo $PATH
#$echo $MANPATH
#$echo $CLASSPATH

Also check if unloading the module restores the environment

#$module unload PROGRAM

Share

Once you are convinced that the module file is working correctly please let your group members also benefit from it. Sharing is easy!

• Move the install directory:

#$mv $HOME/$MY_DISTRO/software/PROGRAM-VERSION  /hpc/local/$MY_DISTRO/$MY_GROUP/software/PROGRAM-VERSION

• Make adjustment in the "local base" line in the module file (see remarks) and move the module file:

#$mv $HOME/$MY_DISTRO/etc/modulefiles/PROGRAM  /hpc/local/$MY_DISTRO/$MY_GROUP/etc/modulefiles/PROGRAM

Allow your group members to add (module files) of other version of the program by making the PROGRAM directory writable:

#$chmod 770 /hpc/local/$MY_DISTRO/$MY_GROUP/etc/modulefiles/PROGRAM

-- Hinri Kerstens - 2016-03-09

Comments