Snakemake can also be used in combination with a High Performance Cluster (HPC). Snakemake can submit jobs and keep track of the running jobs. When there is a dependency between jobs, Snakemake waits for the previous jobs to finish before submitting.
See the following links with information about the execution on a cluster:
Instruction
Running Snakemake
To run snakemake on a cluster the following command can be used:
snakemake --cores 1 --cluster "qsub -q {cluster.q} -N {cluster.N} -M {cluster.M}" --jobs 4 --cluster-config config/cluster.yaml
where:
- The flag `--cluster` the command to sumbit a job to a cluster. With the flag `--cluster-config` additional wildcards for the cluster can be set. In this case the queue (q), job name (N) and email (M)
- The flag `–jobs` specifies the maximum number of jobs.
The snakefile
In the snakefile it is possible to specify which rules are performed locally and which rules are submitted. This can be done with the keyword localrules:
localrules: create_sims, grid, locations
The config file
In a config file it is possible to specify information for submitting, for example the queue and job name. Both JSON and YAML formats are allowed. See below an example in the YAML format:
__default__: q: normal-e5-c7 N: swan M: test.test
Demo files
Link to github: