Editing CU Research Computing (section)

= Slurm =

== Queues==
<syntaxhighlight lang="bash">
#if you want to run on ibg himem, you need to load the right module
module load slurm/blanca

#then in your shell script
#SBATCH --qos=blanca-ibg

#If you want to run on normal queues, then:
module load slurm/slurm

#then in your shell script, one of the below, depending on what queue you want
#SBATCH --qos=himem
#SBATCH --qos=crestone
#SBATCH --qos=janus

</syntaxhighlight>


== Getting information on jobs ==
<syntaxhighlight lang="bash">

#To check our balance on our allocations and get the account id#
sbank balance statement
sacctmgr -p show user <username> #alternatively to find the acct#

#To see how busy the nodes are. For seeing how many janus nodes are available, look for the 
#number under NODES where STATE is "idle" for PARTITION "janus" and TIMELIMIT 1-00:00:00.
sinfo -l  

#checking on submissions for a user
squeue -u <username>  #To see your job statuses (R is for running, PD pending, CG completing, CD completed, F failed, TO timeout)
squeue -u <username> -t RUNNING
squeue -u <username> -t PENDING
squeue -u <username> --start #Get an estimate of when jobs will start

#detailed information on a queue (who is running on it, how many cpus requested, memory requested, time information, etc.)
squeue -q blanca-ibg -o %u,%c,%e,%m,%j,%l,%L,%o,%R,%t | column -ts ','

#current status of queues
qstat -i #To see jobs that are currently pending (this is helpful for seeing if queue is overbooked)
qstat -r #To see jobs that are currently running
qstat -a #To see jobs that are running OR are queued
qstat -a -n #To see all jobs, including which nodes they are running on
qstat -r -n #To see running jobs, and which nodes they are running on

#other commands
showq-slurm -o -U -q <partition>  #List job priority order for current user (you) in given partition
scontrol show jobid -dd <jobid>   #List detailed information for a job (useful for troubleshooting). More info [https://www.rc.colorado.edu/book/export/html/613 here].
pbsnodes -a #To look at the status of each node

### Once job has completed, you can get additional information 
### that was not available during the run. This includes run 
### time, memory used, etc.
sacct -j <jobid> --format=JobID,JobName,MaxRSS,Elapsed     #Stats on completed jobs by jobID
sacct -u <username> --format=JobID,JobName,MaxRSS,Elapsed  #View same info for all jobs of user

#To check graphically how much storage is being taken up in /work/KellerLab folder 
xdiskusage /work/KellerLab/sizes

</syntaxhighlight>

== Running and Controlling jobs ==
<syntaxhighlight lang="bash">
sbatch <shell.script.name.sh> #run shell script
sinteractive --nodelist=bnode0102 #run interactive job on node "bnode0102"
scancel <jobid>                  #Cancel one job
scancel -u <username>            #Cancel all jobs for user
scancel -t PENDING -u <username> #Cancel all pending jobs for user
scancel --name myJobName         #Cancel one or more jobs by name
scontrol hold <jobid>            #Pause job
scontrol resume <jobid>          #Resume job
</syntaxhighlight>

== Advanced commands ==
<syntaxhighlight lang="bash">
### Suspend all running jobs for a user (takes into account job arrays)
squeue -ho %A -t R | xargs -n 1 scontrol suspend
### Resume all suspended jobs for a user
squeue -o "%.18A %.18t" -u <username> | awk '{if ($2 =="S"){print $1}}' | xargs -n 1 scontrol resume
### After resuming, check if any are still suspended
squeue -ho %A -u $USER -t S | wc -l
</syntaxhighlight>

== Interactive Jobs ==
'''Interactive session on IBG himem'''. Note that this command really starts up a job, then launches screen, then connects you to that screen session. If you want to use your emacs key combinations to move the cursor around you'll have to remap in your <tt>~/.screenrc</tt> file the Ctrl-A binding, which in screen is used to initiate commands. This command will bind Ctrl-A to Ctrl-O
 # replace Ctrl-A with Ctrl-O
 escape ^Oo
This only needs to be done once.

Then launch your interactive job on the IBG himem node.
<syntaxhighlight lang="bash">
module load slurm/blanca && sinteractive --qos=blanca-ibg
</syntaxhighlight>

Or onto any free himem node
<syntaxhighlight lang="bash">
module load slurm/blanca && sinteractive --qos=blanca
</syntaxhighlight>

== Job Arrays to submit many independent jobs in parallel ==
The easiest way to do this is to create a 'job array'. This allows you to loop through chromosomes very easily. An example sbatch shell script to tabix index 22 chromosomes is:
<syntaxhighlight lang="bash">
#!/bin/sh                                                                                                                                      
#SBATCH --mem=1gb 
#SBATCH --ntasks-per-node=1     
#SBATCH --nodes=1 
#SBATCH --qos=blanca-ibgc1
#SBATCH --time=2:00:00
#SBATCH --array 1-22 

tabix -fp vcf chr${SLURM_ARRAY_TASK_ID}impv1.vcf.gz
</syntaxhighlight>

== GNU Parallel to submit more complicated sets of jobs in parallel ==
For more complicated sets of commands, one can also create a text file (e.g., test.txt) with the list of parameters you need to feed into your script, for example in bash format. 
<syntaxhighlight lang="bash">
1 0 99
1 100 199
1 200 299
</syntaxhighlight>

You can then feed this text file to the <syntaxhighlight lang="bash" enclose="none">parallel</syntaxhighlight> command. Here's an example for ibgc1 minicluster:
<syntaxhighlight lang="bash">
cat test.txt | parallel --colsep ' ' sbatch --qos blanca-ibgc1 test.sh
</syntaxhighlight>

Here's a command for janus
<syntaxhighlight lang="bash">
cat test.txt | parallel --colsep ' ' sbatch --qos janus --reservation=janus-serial test.sh
</syntaxhighlight>


Where the file test.sh is a bash script that simply runs tabix on the commands from the test.txt file.

<syntaxhighlight lang="bash">
#!/bin/sh
#SBATCH -A <account number>
#SBATCH --mem=20gb 
#SBATCH --ntasks-per-node=1
#SBATCH --nodes=1
#SBATCH --time=0:30:00

assertNotEmpty() {
    : "${!1:? "$1 is empty, aborting."}"
}
: ${chr:=$1}
export chr
assertNotEmpty chr

: ${startpos:=$2}
export startpos
assertNotEmpty startpos

: ${endpos:=$3}
export endpos
assertNotEmpty endpos

tabix -h chr${chr}/chr${chr}impv1.vcf.gz ${chr}:${startpos}-${endpos} | bgzip -c > chr$chr/chr${chr}impv1.${chr}_$startpos-$endpos.vcf.gz
</syntaxhighlight>