How to submit a SLURM job via sbatch that depends on a list of jobs

Question

I have a shell script that submits multiple jobs to SLURM. These jobs are submitted within the following loop:

nb_partitions=72
slurmids=() # storage of slurm job ids
for k in $(seq 1 $nb_partitions); 
do
      cd results/partition$k/MainFolder
      ID=$(sbatch --parsable estimation.sh)
      slurmids+=($ID)
      cd ..
      cd ..
      cd ..
done
echo "Jobs are now running."

In addition to submitting the jobs, this loop also creates the array slurmids that contains a list of the job ids for all the SLURM jobs.

Now, I have another SLURM job that I would like to submit with sbatch YY.sh, but this job needs to wait to be submitted until all of the previous jobs finish. How can I do this? The command that comes to mind is sbatch --dependency=afterok<jobID1:jobID2:...:jobID72> YY.sh, I'm not sure how to use my slurmids list in the afterok statement.

Any help would be much appreciated!

Answer 1

The --dependency syntax is:

afterok:job_id[:jobid...]

So you need your array separated by a colon. You could simply echo the whole list and replace the spaces by colons in your command:

sbatch --dependency=afterok:$(echo ${slurmids[*]} | tr ' ' :) YY.sh

If you just need the slurmids for that single purpose, you could concatenate them in your first script in this way:

[...]
slurmids=""
for k in $(seq 1 $nb_partitions); 
do
      cd results/partition$k/MainFolder
      slurmids="$slurmids:$(sbatch --parsable estimation.sh)"
[...]

This will result in the a $slurmids string with all the IDs separated with : and a leading :, so you could send:

sbatch --dependency=afterok$slurmids YY.sh