1

I am trying to run multiple (several hundred) very similar job-files with slurm using sbatch.

My .job file look like:

#SBATCH ...
...
...
srun ./someProg -a A -b B -c C -d D

Is there any convenient way to submit the job file using sbatch with multiple options for A/B/C and D and generating a new job for every combination of A/B/C/D, without just generating hundreds of .job files? (I have already seen a lot of arrays in slurm files, but I do not think it will help me here anyway.)

iiiiiiiiiiiiiiiiiiii
  • 319
  • 3
  • 13

3 Answers3

1

without just generating hundreds of .job files?

You can use bash's Process Subsitution for replacing the creation of files:

#!/bin/bash

genjob() {
    local content
    IFS='' read -d '' -r content <<-EOF
        #!/bin/bash
        #SBATCH ...
        ...
        ...
        srun ./someProg $(printf '%q ' "$@")
    EOF
    printf '%s\n' "$content"
}

sbatch <(genjob -a A -b B -c C -d D)

important: The dash in <<-EOF means that the TAB characters at the beginning of each line of the Heredoc will be stripped; so the indentation must be done with TABs.

Fravadona
  • 13,917
  • 1
  • 23
  • 35
1

You say you have several hundred jobs like that. That may be more than your available core count, so you want to be careful how you submit it. You want to submit as many as possible, but not all of them at once.

Here are two utilities that accept an arbitrary long list of commandlines, and then spread them over the available nodes/cores:

https://github.com/TACC/launcher

https://github.com/TACC/pylauncher

Victor Eijkhout
  • 5,088
  • 2
  • 22
  • 23
1

I often pass the necessary information as arguments to the job file: The contents of the submit file (let's call it submit.sh) may look like this:

#SBATCH ...
...
...
srun ./someProg -a $1 -b $2 -c $3

In a second bash script i loop over all options that I need. The bash script could look like this:

#!/bin/bash

for aa in a1 a2 a3
do
  for bb in b1 b2 b3
  do
    for cc in c1 c2 c3
    do
      scommand="sbatch -J A=${aa},B=${bb},C=${cc} submit.sh $aa $bb $cc"
      echo "submit command: $scommand"
      $scommand
    done
  done
done

The -J option sets the name of the slurm job.

nameiki
  • 138
  • 9