submit sbatch jobs in loop

Question

I have a txt file (say jobs.txt) that has several lines like:

"sbatch -w node00x script.sh 1"
"sbatch -w node00z script.sh 10"
.
.
etc

I wonder if it is possible to create an executable bash file like the following

#!/bin/bash

while read -r line;
do submit the job by line;
done < jobs.txt;

that I can just execute and which will run the jobs in respective nodes. I have very limited knowledge in this area. Would appreciate any help.

Is there a strong reason why you specify manually the compute nodes on which you want to run each job? Also I suspect your use case really fits a job array ; if that's the case, you could greatly simplify your whole process. — damienfrancois, Nov 30 '18 at 14:58
@damienfrancois- Yes. I need very specific nodes. My script.sh has job-array set up for any particular node which will be queued. You are right though. I also think, even this can be achieved by carefully creating job-array setup. But I am no expert in this. — Ghosh, Dec 01 '18 at 16:55

score 2 · Answer 1 · answered Nov 29 '18 at 10:56

2

If I understand you correctly, you just need to do eval command:

eval $line

answered Nov 29 '18 at 10:56

Gustavo Ferreira

this worked! The only thing I had to do is to remove the quotes. – Ghosh Nov 30 '18 at 06:17
1

Use tr to remove the quotes. – Gustavo Ferreira Nov 30 '18 at 09:39

Bub Espinja · Answer 2 · 2018-11-29T12:11:35.273

2

What I usually do in those cases is:

./jobs.txt

If you have set the execution permissions for that file (as its extension suggests), give them to it:

chmod u+x jobs.txt

And if it is necessary, I would add to the head of the jobs.txt:

#!/bin/bash

And if you want to remove any confusion with the file name, rename it:

mv jobs.txt jobs.sh

Now, you have a workload ready to be submitted!

edited Nov 29 '18 at 12:11

answered Nov 29 '18 at 11:59

Bub Espinja

thanks for the nice solution! Basically it avoided creating an extra launcher script. – Ghosh Nov 30 '18 at 06:19

2 Answers2