SoftwareInstallation < HPC

Tags: view all tags
---+!! %TOPIC%

%TOC%

---++ Can I install my own software?

The answer is: yes, (of course) you can. There is a [[HPCInfrastructure#Directories][group-specific directory]] that is made available for this purpose:
<verbatim>/hpc/local/osversion/group/</verbatim>

Note that =osversion= is the current Linux version and =group= is your own group name. Here, you can install any software you like and maintain it yourself. See [[#How_to_install_group_specific_so][here]] for explanations how to install C, Perl or R packages.

---++ Software of general interest

If you find a software to be of general interest to HPC users, let [[ContactDetails][us]] know. We can provide is as an software module or install it as a system package (rpm) and update it on a regular basis.

However, if you are dependent on a specific version of a software package and don't want regular updates, we advise you to [[#How_to_install_group_specific_so][install it yourself]] and encourage you to take the benefits of [[#Making software available using LMOD]].

---++ How to install your own software

A specific directory is made available for group-specific software to be installed:
<verbatim>/hpc/local/osversion/group/</verbatim>

---+++ C software

You can download your C software of interest and unpack it in this directory. Typically, a pre-installation configuration is done by executing:
<verbatim>./configure --prefix /hpc/local/osversion/group/package</verbatim>

This will create a "Makefile" which explains to the "make" utility how the software should be compiled, and where the software will be installed. The 'prefix' is the top directory under which the whole package will be installed. You may want to include the package version number in the name of this directory; that way you can install more than one version next to each other. Use symlinks to point to the 'default' installation.

After the =configure= step, you compile the software using:
<verbatim>make</verbatim>

The Makefile will be read, building the application binaries. To install these binaries, use:
<verbatim>make install</verbatim>

That is it! You can check the user documentation of the installed software for details of how to run the application.

---+++ Python modules 
use pip to install python modules.<br/>
=module load python= <br/>
=module list= <br/>
=pip list= <br/>
=pip install --user &lt;module&gt;= <br/>
<br/>
Create your virtual software environment for a specific Python project with =Python Virtual Environment= : <br />
=see :=  http://docs.python-guide.org/en/latest/dev/virtualenvs 
<br/>

---+++ Your own R version

Go to https://cran.r-project.org/mirrors.html, download the latest "R-3.0.whatever.tar.gz" file to a submit host (hpcs01/hpcs02).
Copy this file to a directory that has enough space (about 300M). In this case, let's assume /tmp, but your homedir probably has enough space as well.

<pre>cd /tmp
wget http://cran-mirror.cs.uu.nl/src/base/R-3/R-3.0.2.tar.gz
tar -zxvf R-3.0.2.tar.gz
cd R-3.0.2
</pre>

Now, we'll have to "configure" and "make" this:

<pre>./configure --prefix=/hpc/local/CentOS7/bofh/R-3.0.2
make
make install
</pre>

Where you replace "/hpc/local/CentOS7/bofh/R-3.0.2" with some path where you have write-access. Something like "/hpc/local/CentOS7/YOURGROUP/R-3.0.2" would probably be a good choice.

Now, you would start this version of R by entering the full path:

<pre>/hpc/local/CentOS7/bofh/R-3.0.2/bin/R</pre>

Or by adding the directory /hpc/local/CentOS7/YOURGROUP/R-3.0.2/bin to your own PATH (e.g. in your $HOME/.bash_profile).

---+++ R packages

Packages can easily be installed inside R by providing a local path:

<verbatim>
R
install.packages( "yourLibrary", lib = "/hpc/local/osversion/group/path" )
library( "yourLibrary", lib.loc = "/hpc/local/osversion/group/path" )</verbatim>

Alternatively, you can [[#How_to_customize_your_Linux_envi][customize]] your Linux environment variables and set =R_LIBS= to =/hpc/local/osversion/group/path=. This way, you can leave out the path specification in =R=.

*Check* <br/>
=module load R=  <br/>
=R=  <br/>
=find.package('&lt;mypackage&gt;')=  <br/>



---+++ Perl libraries

To install CPAN perl libraries, you first have to instruct CPAN which directory to use. You can do this by modifying your CPAN configuration, from within the CPAN shell:

<verbatim>
cpan
o conf mbuildpl_arg "installdirs=site install_base=/hpc/local/osversion/group"
o conf makepl_arg "INSTALLDIRS=site INSTALL_BASE=/hpc/local/osversion/group"
o conf prefer_installer MB
o conf prerequisites_policy follow
o conf commit</verbatim>

After this, CPAN should install all perl libraries in the appropriate directory such that they are available on the entire cluster. Your =PERLLIB= environment variable should be set to include =/hpc/local/OSVERSION/GROUP/lib/perl5=.

In the past, the environment variable =PERL_INSTALL_ROOT= used to be used for cpan installs, but that doesn't work anymore and wreaks havoc on the install if you use the cpan configuration shown above. I.o.w., be sure not to define =PERL_INSTALL_ROOT=.

If you later just wish to install packages, you can use the =cpan -i <module>= command from the command line. No need to startup the CPAN shell.

---+++ Making software available using =GUIX=
=Guix= is a package manager and is installed for general use. <br> 
   * [[https://github.com/UMCUGenetics/guix-documentation/blob/master/for-scientists/guix-for-scientists.md][GNU Guix for scientists]]. <br />
   * [[https://github.com/UMCUGenetics/guix-documentation/blob/master/for-bioinformaticians/guix-for-bioinformaticians.md][GNU Guix for bioinformaticians]].<br />
   * [[https://github.com/UMCUGenetics/guix-documentation/blob/master/solutions/common-problems-and-solutions.md][Common problems and solutions ]].<br />
You can find information in the 
   * [[https://en.wikipedia.org/wiki/GNU_Guix][Wiki]] . <br />
and 
   * [[https://hpcguix.op.umcutrecht.nl/getting-started][Guix website]] in the UMC-DBG domain . <br />

*Add these lines in your ~/.bash_profile file to enable guix* .

<verbatim>
# guix environment 
# see : https://hpcguix.op.umcutrecht.nl/getting-started 

export PATH=$PATH:"/gnu/profiles/base/bin"

GX_PROFILE="$HOME/.guix-profile"
if [ -f "$GX_PROFILE/etc/profile" ]; then
  . "$GX_PROFILE/etc/profile"
fi
</verbatim>


---+++ Making software available using =LMOD=
[[#Making software available using LMOD]].


---+++ Making software available using =SINGULARITY=
[[https://www.sylabs.io/docs][Singularity]]  (docker compatible but more secure) is available on the compute-nodes (n00XX).
<br/>
You can download docker- or singularity- images and run your programs in singularty.<br/> 
p.e.  R and SAIGE module :<br/> 
<br/>
You can execute this in a qlogin session.<br/>

*SAIGE on singularity(image)* <br/>
# Download a singularty image for SAIGE:<br/>
=singularity pull shub://singularity-hub.org/statgen/singularity-saige= <br/>

# Login into the container<br/>
=singularity shell ./statgen-singularity-saige-master-latest.simg= <br/>
<br/>
# Check if SAIGE is available
<verbatim>
R
installed.packages() 
q()
n
</verbatim>
<br/>
# run a R script in the container<br/>
# create 1.R<br/>
<verbatim>
library("SAIGE") 
print("Hello World!") 
</verbatim>

# You can run this in a qsub script <br/>
# Run your code by : <br/>
=singularity exec ./saige-0.35.8.1.simg Rscript 1.R= <br/>

*SAIGE on singularity with docker image* <br/>
<verbatim>
singularity pull docker://wzhou88/saige:0.35.8.1
singularity shell ./saige:0.35.8.1
R
installed.packages()
q()
n
</verbatim>

# run <br/>
=singularity exec ./saige-0.35.8.1.simg Rscript 1.R= <br/>
Topic revision: r16 - 2020-08-10 - ReneJanssen