More Slurm Info

• srun -c 4 bash -c "hostname; stress -c 10". This will start 1 task, getting 4 cores (2 CPU's, 2 cores on each).

• srun -n 4 bash -c "hostname; stress -c 10". This will start 4 seperate "tasks", each getting 1 CPU (2 cores on each). Eight threads in total.

The previous form (-c) is usually what you want. Just one "job" with 4 CPU cores.

• srun -c 2 --mem 5G --time 01:00:00 --pty bash

• srun -c 2 --mem 5G --time 01:00:00 --nodelist n0014 --pty bash

FAQ about SLURM

• There are two "partitions" (like a "queue" in GridEngine), called "cpu" (for normal jobs) and "gpu" (where some GPU's are available). The default is "cpu".

will request a specific type of GPU.

Using a GPU

Something like:

srun -n 12 --gres=tmpspace:10G --gres=gpu:1 --time 24:00:00 --mem 100G --pty bash

will give you an interactive session with 1 GPU.

• By default, your job gets 1 GB of "scratch" local disk space in "$TMPDIR" (where TMPDIR=/scratch/$SLURM_JOB_ID).

salloc/srun

Quoting from the documentation:

The final mode of operation is to create a resource allocation and spawn job steps within that allocation. The salloc command is used to create a resource allocation and typically start a shell within that allocation. One or more job steps would typically be executed within that allocation using the srun command to launch the tasks. Finally the shell created by salloc would be terminated using the exit command.

Be very careful to use srun to run the commands within your allocation. Otherwise, the commands will run on the machine that you're logged in on! See:

# Allocate two compute nodes:
[mmarinus@hpcm05 ~]$ salloc -N 2
salloc: Pending job allocation 1635
salloc: job 1635 queued and waiting for resources
salloc: job 1635 has been allocated resources
salloc: Granted job allocation 1635
salloc: Waiting for resource configuration
salloc: Nodes n[0009-0010] are ready for job

# I got n0009 and n0010
[mmarinus@hpcm05 ~]$ srun hostname
n0009.compute.hpc
n0010.compute.hpc

# But this command just runs on the machine I started the salloc command from!
[mmarinus@hpcm05 ~]$ hostname
hpcm05.manage.hpc

# Even if you "srun" something, be careful where (e.g.) variable expansion is done:
[mmarinus@hpcm05 ~]$ srun echo "running on $(hostname)"
running on hpcm05.manage.hpc
running on hpcm05.manage.hpc

# Exit the allocation
[mmarinus@hpcm05 ~]$ exit
exit
salloc: Relinquishing job allocation 1635

Local (scratch) disk space

If your job benefits from (faster) local disk space (like "qsub -l tmpspace=xxx"), request it like this:

srun --gres=tmpspace:250M --pty bash

Of course, this works for all the commands. The scratch disk space will be made available in $TMPDIR (/scratch/$SLURM_JOB_ID) and will be erased automatically when your job is finished.

SGE versus SLURM

SGE versus SLURM

SGE	vs.	SLURM
qstat	-	squeue
qsub	-	sbatch
qdel	-	scancel
qlogin	-	salloc

examples :
submit job named 1.sh
qsub 1.sh sbatch 1.sh

Slurm

Starting in May 2019, we're testing our new Slurm setup. Slurm is similar to GridEngine: it manages a cluster, distributing user jobs in (hopefully) a fair and efficient way.

The concepts are comparable, but the syntax is not.

This page will hopefully grow organically. Feel free to make corrections and add your tips, tricks and insights.

Defaults

Some defaults:

• There is one "partition" (like a "queue" in GridEngine). It is called "one" (suggestions welcome; it cannot be called "default").
• Default runtime is 10 minutes.
• Default memory is 10 GB.
• By default, your job gets 1 GB of "scratch" local diskspace in "$TMPDIR".

Running jobs

You can run jobs using "srun" (interactively) or "sbatch" (like qsub).

srun

srun will execute the command given, and wait for it to finish. Some examples:

• srun sleep 60

• srun -n 4 bash -c "hostname; stress -c 10". This will start 4 seperate "tasks", each getting 1 CPU (2 cores on each). Eight threads in total.

This is different from:

• srun -c 4 bash -c "hostname; stress -c 10". This will start 1 task, getting 4 cores (2 CPU's, 2 cores on each).

To me, the number of tasks, CPU's and cores is sometimes slightly surprising. I guess it will make sense after a while...

You can also use srun to get an interactive shell on a compute node (like qlogin):

• srun -n 2 --mem 5G --time 01:00:00 --pty bash

Or on a specific node:

• srun -n 2 --mem 5G --time 01:00:00 --nodelist n0014 --pty bash

sbatch

sbatch is like qsub. Commandline options are similar to srun, and can be embedded in a script file:

#!/bin/bash

#SBATCH -t 00:05:00
#SBATCH --mem=20G
#SBATCH -o log.out
#SBATCH -e errlog.out
#SBATCH --mail-type=FAIL
#SBATCH --mail-user=youremail@some.where #Email to which notifications will be sent

env
echo "Hello World" 

-- Martin Marinus - 2019-05-07

Comments

<--/commentPlugin-->