Slurm is used for cluster management and job scheduling. Slurm has three key features:

1. Allocates resources to users
2. Provides a framework for starting, executing, and monitoring work
3. Manages a queue of pending work

Sbatch

Submit a bash script to slurm. Slurm will schedule this bash script given the arguments presented in the script.

$ sbatch script.sh

Script example:

#! /bin/bash
#################### Batch Headers ####################
#SBATCH -p drcluster        # Get it? DRC cluster ;)
#SBATCH -J hello_world      # Custom name
#SBATCH -o results-%j.out   # stdout/stderr redirected to file
#SBATCH -N 3                # Number of nodes
#SBATCH -n 1                # Number of cores (tasks) 
#######################################################

python hello_world.py

Arguments:

• -N --nodes: Number of nodes (machines) to allocate.
• -n --ntasks: Number of cores to be allocated. Cores may or may not come from the same machine. Defaults to one task per node.
• --ntasks-per-node: Specify the number of tasks per node to use. --ntasks takes precedence.
• -p --partition: Specify partition to run on.
• --mem: Amount of memory to reserve (G for gigabytes)
• -J: Job name (for sacct or squeue)
• -o: Output file name (%j is job number)
• --nodelist: Request specific nodes. Comma separated node list or a filename (should have '/' in the name)

Environment variables:

• $SLURM_SUBMIT_DIR: The directory you submitted the job from. Might want to cd to here at the start of the script. Should be a place where all of the nodes can access (shares storage).

Scancel

scancel is used to cancel a task

$ scancel <jobID>

Scontrol

scontrol allows you to view or alter a job's details

View job details:

$ scontrol show job 14

Suspend a job:

$ sudo scontrol suspend 14

Continue a job:

$ scontrol resume 14

Give up on a job:

$ scontrol release 14

Sinfo

sinfo provides information about the cluster

$ sinfo
PARTITION  AVAIL  TIMELIMIT  NODES  STATE NODELIST
drcluster*    up   infinite      3   idle node[02-04]

• PARTITION is the cluster partition name
• TIMELIMIT is the maximum amount of time a user can request resources for this partition
• NODES is the number of compute nodes (clients) on this partition
• STATE
• MIXED: Some CPUs are ALLOCATED (used) while others are IDLE
• RESERVED: Noe is reserved in advanced

Squeue

squeue displays all submitted jobs.

$ squeue
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
   22 drcluster hostname    drc14 PD       0:00      3 (PartitionConfig)

A list of state codes can be found HERE.

Srun

srun allows you to run parallel jobs directly from the command line. See sbatch for command line arguments.

$ srun --nodes=3 hostname
node03
node02
node04

To run an interactive srun session:

$ srun --pty /bin/bash