CU Research Computing

General documentation for using RC is [on their website]. This document will mostly cover specific instructions for using it in the Vrieze and Keller labs.

Logging in

Put these settings in your ~/.ssh/config file so you only have to enter your OTP once per session, instead of for every ssh connection you make

# These rules only apply for connections to login.rc.colorado.edu
Host login.rc.colorado.edu
# Setup a master ssh session the first time, and subsequent times
# use the existing master connection
ControlMaster auto
ControlPath ~/.ssh/%r@%h:%p
# X forwarding. Remove this on a Mac if you
# don't want it to start X11 each time you
# connect to RC
ForwardX11 yes
# Compress the data stream.
Compression yes
# Send keep alive packets to prevent disconnects from
# CU wireless and behind NAT devices
ServerAliveInterval 60

These settings should work from Mac and Linux. I'm not sure how to do the equivalent from Windows with Putty. On a Mac, those settings will cause X11 to start. If you don't want that to happen, then remove the ForwardX11 yes line.

Slurm

Getting information on jobs

squeue -u <username>
squeue -u <username> -t RUNNING
squeue -u <username> -t PENDING
showq-slurm -o -U -q <partition>  #List job priority order for current user (you) in given partition
scontrol show jobid -dd <jobid>   #List detailed information for a job (useful for troubleshooting)

### Once job has completed, you can get additional information 
### that was not available during the run. This includes run 
### time, memory used, etc.
sacct -j <jobid> --format=JobID,JobName,MaxRSS,Elapsed     #Stats on completed jobs by jobID
sacct -u <username> --format=JobID,JobName,MaxRSS,Elapsed  #View same info for all jobs of user

Controlling jobs

scancel <jobid>                  #Cancel one job
scancel -u <username>            #Cancel all jobs for user
scancel -t PENDING -u <username> #Cancel all pending jobs for user
scancel --name myJobName         #Cancel one or more jobs by name
scontrol hold <jobid>            #Pause job
scontrol resume <jobid>          #Resume job

Advanced commands

### Suspend all running jobs for a user (takes into account job arrays)
squeue -ho %A -t R | xargs -n 1 scontrol suspend
### Resume all suspended jobs for a user
squeue -o "%.18A %.18t" -u <username> | awk '{if ($2 =="S"){print $1}}' | xargs -n 1 scontrol resume
### After resuming, check if any are still suspended
squeue -ho %A -u $USER -t S | wc -l

Interactive Jobs

Interactive session on IBG himem. Note that this command really starts up a job, then launches screen, then connects you to that screen session. If you want to use your emacs key combinations to move the cursor around you'll have to remap in your ~/.screenrc file the Ctrl-A binding, which in screen is used to initiate commands. This command will bind Ctrl-A to Ctrl-O

# replace Ctrl-A with Ctrl-O
escape ^Oo

This only needs to be done once.

Then launch your interactive job.

module load slurm/blanca && sinteractive --qos=blanca-ibg

Job Arrays to submit many independent jobs in parallel

The easiest way to do this is to create a 'job array'. This allows you to loop through chromosomes very easily. An example sbatch shell script to tabix index 22 chromosomes is:

#!/bin/sh                                                                                                                                      
#SBATCH --mem=1gb 
#SBATCH --ntasks-per-node=1     
#SBATCH --nodes=1 
#SBATCH --qos=blanca-ibgc1
#SBATCH --time=2:00:00
#SBATCH --array 1-22 

tabix -fp vcf chr${SLURM_ARRAY_TASK_ID}impv1.vcf.gz

GNU Parallel to submit more complicated sets of jobs in parallel

For more complicated sets of commands, one can also create a text file (e.g., test.txt) with the list of parameters you need to feed into your script, for example in bash format.

1 0 99
1 100 199
1 200 299

You can then feed this text file to the parallel command. Here's an example for ibgc1 minicluster:

cat test.txt | parallel --colsep ' ' sbatch --qos blanca-ibgc1 test.sh

Here's a command for janus

cat test.txt | parallel --colsep ' ' sbatch --qos janus --reservation=janus-serial test.sh

Where the file test.sh is a bash script that simply runs tabix on the commands from the test.txt file.

#!/bin/sh
#SBATCH -A <account number>
#SBATCH --mem=20gb 
#SBATCH --ntasks-per-node=1
#SBATCH --nodes=1
#SBATCH --time=0:30:00

assertNotEmpty() {
    : "${!1:? "$1 is empty, aborting."}"
}
: ${chr:=$1}
export chr
assertNotEmpty chr

: ${startpos:=$2}
export startpos
assertNotEmpty startpos

: ${endpos:=$3}
export endpos
assertNotEmpty endpos

tabix -h chr${chr}/chr${chr}impv1.vcf.gz ${chr}:${startpos}-${endpos} | bgzip -c > chr$chr/chr${chr}impv1.${chr}_$startpos-$endpos.vcf.gz

KellerLab software

These instructions are for accessing the software and utilities that have been installed by the KellerLab. There are two choices.

Source the KellerLab preferences

This will import all of the KellerLab preferences, including changing the prompt and adding some other useful aliases. To do this add the following to the bottom of your .my.bashrc

. /work/KellerLab/opt/bash_profile

an easy way to do that is by pasting the following into a terminal on RC

echo ". /work/KellerLab/opt/bash_profile" >> ~/.my.bashrc

Only add the paths

If you prefer your own command prompt and other settings, then you can just add the paths to your .my.bashrc To do that, add the following lines to the bottom of your .my.bashrc

PATH=${PATH}:/work/KellerLab/opt/bin 
MANPATH=${MANPATH}:/work/KellerLab/opt/share/man
export PERL5LIB=/work/KellerLab/opt/lib/perl5/

Emacs and ESS

ESS on RC

To add the ESS package to Emacs, add the following lines to the bottom of your .emacs file

;;getting ESS to work
(load "/work/KellerLab/opt/ESS/ess-13.09-1/lisp/ess-site.el")

(require 'ess-site)

ESS on your local Emacs

Running Emacs (or Aquamacs) on your local computer, but editing files and using R on RC is not difficult, but requires some setup.

Setup

First, edit your local .emacs file and add the following statement

(setq tramp-ssh-controlmaster-options
                 (concat
                   "-o ControlPath=~/.ssh/%%r@%%h:%%p "
                   "-o ControlMaster=auto -o ControlPersist=no"))

This will cause tramp to share the same SSH socket as your login.rc.colorado.edu SSH sessions. That way you can open files without having to re-enter your OTP.

Opening a remote file with tramp

Opening a file on a remote host with tramp is similar to opening a local file. Press ctrl-x ctrl-f and then for the remote file enter /ssh:login.rc.colorado.edu: and then enter the path to the file, for example /ssh:login.rc.colorado.edu:/work/KellerLab/myfile.R or /ssh:login.rc.colorado.edu:scripts/mystuff.R. Once a single path element (/work or ~/, for example> is entered then tab completion will work.

Activating a remote R session in ESS

Activating a remote R session is accomplished by using a shell within Emacs. It is most convenient to open a new Emacs frame with ctrl-x 5 2. Then within the new frame enter M-x shell. Then from the shell within Emacs run ssh login.rc.colorado.edu and then from that shell run ssh himem04 or ssh node0219 or whatever to get to the location where you actually want to run R.

Once you have a shell on your job node, then run R by typing R. Then run M-x ess-remote and then press enter to accept the Dialect line that is put up.

Once that is done you will be able to edit an R script in your local Emacs and submit lines to the R process on the RC job node.