5

I have a long-running task that I want to run using a job array on slurm.

The script I am currently using to submit the jobs is:

#!/bin/bash

#SBATCH --output=slurm-%A_%a.out

#SBATCH --array=1-30

#SBATCH --ntasks=1

#SBATCH --qos=qos-15d

#SBATCH --partition=large

#SBATCH --mem=4G

srun ./a

This script works fine, but my problem is as it is an array of 30 jobs, I need to start the first one at time X and then start the second after X minutes and so on. I want to do this because I will simulate a C-compiled program that uses the srand (time (0)) function to generate random numbers. Therefore, the above script produces the same results for 30 simulations, because the random number generated will be equal. As each simulation takes a long time to run, it is not feasible for me to wait for a job to complete before starting another job.

Amália Vasconcelos
  • 53
  • 3

2 Answers2

3

I know that this issue already has an answer, but after I had the same problem I found an alternative way to solve it.

The original answer had the problem for me that there is a chance that multiple runs would start at the same time since the delay is random and not unique.

My solution uses environment variables most slurm systems provide in this case specifically $SLURM_ARRAY_TASK_ID. (More environment variables can be found here: https://slurm.schedmd.com/sbatch.html#OPT_SLURM_ARRAY_TASK_ID)

For the use case above the script would look something like this. Where the number in the sleep statement can be the wanted delay.

#!/bin/bash

#SBATCH --output=slurm-%A_%a.out
#SBATCH --array=1-30
#SBATCH --ntasks=1
#SBATCH --qos=qos-15d
#SBATCH --partition=large
#SBATCH --mem=4G

sleep $((SLURM_ARRAY_TASK_ID*5))

srun ./a

This will guarantee separate starting times of the program.

It would also be an option the read in the $SLURM_ARRAY_JOB_ID and $SLURM_ARRAY_TASK_ID into the program and use these as the seed for the random number generator for the simulation. (Reading in Environment Variables in C http://www0.cs.ucl.ac.uk/staff/W.Langdon/getenv/)

ClemensFricke
  • 31
  • 2
  • 2
2

Assuming you are the only one using the cluster (otherwise, startup times will not be the same for all your jobs), one little trick is to add a random sleep at the beginning of your script:

#!/bin/bash

#SBATCH --output=slurm-%A_%a.out
#SBATCH --array=1-30
#SBATCH --ntasks=1
#SBATCH --qos=qos-15d
#SBATCH --partition=large
#SBATCH --mem=4G

sleep $((RANDOM%30+1))

srun ./a

Even if all jobs start at the same time, they will spend some random time (from 1 to 30 seconds) sleeping before actually starting the computations.

damienfrancois
  • 52,978
  • 9
  • 96
  • 110