Master SLURM Script that calls other SLURMS

Question

I'm a cancer researcher and we were wondering if any of you knew how to make a SLURM script that calls other slurms. If the previous script finished successfully, continue to the next call. I.E something like:

#!/bin/bash
#SBATCH --nodes=1 
#SBATCH --tasks-per-node=1
#SBATCH --time=24:00:00
#SBATCH --mem=40GB?
#SBATCH --job-name=Master_script
#call first slurm
sbatch a.slurm
#if it completes successfully
if a.slurm:
   sbatch b.slurm
   if b.slurm:
      sbatch c.slurm
      sbatch d.slurm
      if c.slurm:
          sbatch e.slurm
      else:
         echo "c.slurm did not complete successfully"
   else:
      echo "b.slurm did not complete successfully"
else:
    echo "a.slurm did not complete successfully"

score 4 · Answer 1 · answered Dec 07 '18 at 08:54

You can submit the 5 jobs with the --dependency option of sbatch.

From the manpage:

-d, --dependency= Defer the start of this job until the specified dependencies have been satisfied completed.

In your case, it would go like this:

A=$(sbatch --parsable a.slurm)
B=$(sbatch --parsable --dependency=afterok:$A a.slurm)    
C=$(sbatch --parsable --dependency=afterok:$B a.slurm)    
D=$(sbatch --parsable --dependency=afterok:$B a.slurm)    
E=$(sbatch --parsable --dependency=afterok:$D a.slurm)

Note that in such a case jobs that depend on other jobs will stay pending indefinitely if the dependent job fails. You will either have to clean them manually or submit empty jobs for the cases where a job fails with --dependency=afternotok:....

If your workflow gets more complicated, it could be interesting to investigate the use of workflow managers such as Bosco or Fireworks.

Master SLURM Script that calls other SLURMS

1 Answers1