1

I usually run two separate jobs (program1 and program2) on two different GPUs.

I would like to be able to run these two jobs from a single bash script but still on two different GPUs with a slurm .out file for each programs. Is this possible?

#!/bin/bash -l
#SBATCH --time=1:00:00
#SBATCH --gres=gpu:v100:1
#SBATCH --mem=90g
#SBATCH --cpus-per-task=6 -N 1

program1 

#!/bin/bash -l
#SBATCH --time=1:00:00
#SBATCH --gres=gpu:v100:1
#SBATCH --mem=90g
#SBATCH --cpus-per-task=6 -N 1

program2

The script below seems to run both programs on the same GPU with a single .out file as output.

#!/bin/bash -l
#SBATCH --time=1:00:00
#SBATCH --gres=gpu:v100:1
#SBATCH --mem=90g
#SBATCH --cpus-per-task=6 -N 1

program1 &
program2 &
wait

Thanks for your help.

Tim
  • 513
  • 5
  • 20

1 Answers1

1

First way

You could write a submit script that gets the name of the executable as a command line argument and another script that calls the submit script. The submit script "submit.sh" could look like this:

#!/bin/bash -l
#SBATCH --time=1:00:00
#SBATCH --gres=gpu:v100:1
#SBATCH --mem=90g
#SBATCH --cpus-per-task=6 -N 1

$1

The second script "run_all.sh" could look like this:

#!/bin/bash

sbatch submit.sh program1
sbatch submit.sh program2

Now you can start your jobs with:$ ./run_all.sh

Second way

You don't have to use scripts to provide all the information for slurm. It is possible to pass the job information as arguments from the sbatch call: sbatch [OPTIONS(0)...] [ : [OPTIONS(N)...]] script(0) [args(0)...]

A script like this then could be useful:

#!/bin/bash -l
slurm_opt= --time=1:00:00 --gres=gpu:v100:1 --mem=90g --cpus-per-task=6 -N 1 --wrap
sbatch $slurm_opt program1
sbatch $slurm_opt program2

Note the --wrap option. It allows you to have any executable not just a script after it.

nameiki
  • 138
  • 9
  • Thanks for your suggestion. Isn't there a way to do this from a single .sh file and have separate slurm.out files for each task? – Tim Aug 29 '21 at 13:03
  • I updated the answer with a way that only uses one script. It is a little bit less flexible but it can easily be used for simple job submissions. – nameiki Aug 30 '21 at 06:51