BSUB many matlab job to a cluster?

Question

I am using the following bash file to submit matlab job to a cluster,

#!/bin/bash
#BSUB -L /bin/bash
#BSUB -J matlab.01
#BSUB -q long
#BSUB -n 32
#BSUB -R "span[hosts=1]"
#BSUB -W 20:00
#BSUB -R "rusage[mem=3072]"

#BSUB -o %J.out
#BSUB -e %J.err

# the working directory
work=/home/models
cd $work

# run matlab on the main function
matlab -logfile ./output.txt -nodisplay -r "foo('model', day);"

suppose the file name is mat.bash, then I use the command

bsub < mat.bash

to submit ONE job to the cluster. The last line in the bash file include a function

fool(model, day)

In this function, model will have four alternatives, and day will have 200 alternatives, which means I have 4 X 400 = 800 jobs to submit to the cluster, each job will run about 16 hours.

what is the most convenient way to submit the 800 jobs, not submit one by one?

The goal is to have multiple jobs can be running on the cluster at the same time, do not need to wait one job finished than start another one.

Thanks in advance!

score 1 · Answer 1 · answered Dec 17 '15 at 16:26

My suggestion would be to remove the actual call to matlab from your script, then write a separate script to iterate over the possible values of 'model' and 'day', append the appropriate matlab call, and submit each job for you.

Something like:

#!/bin/sh

for model in one two three four
do
    for day in `seq 200`
    do  
        cp mat.bash mat.bash.$model.$day
        echo "matlab -logfile ./output.txt -nodisplay -r \"foo('$model', $day);\"" >> mat.bash.$model.$day
        bsub < mat.bash.$model.$day
        rm mat.bash.$model.$day
    done
done

score 1 · Answer 2 · answered Dec 18 '15 at 02:11

You could try using a job array. Decompose the array index to get the model and day parameters. Something like this:

#!/bin/bash
#BSUB -L /bin/bash
#BSUB -J matlab.01[1-6]
#BSUB -R "span[hosts=1]"

#BSUB -o %J-%I.out
#BSUB -e %J-%I.err

PARAM1=$(((LSB_JOBINDEX-1)%2))
PARAM2=$(((LSB_JOBINDEX-1)/2))
echo "PARAM1=$PARAM1"
echo "PARAM2=$PARAM2"

Then you can submit all jobs with a single bsub

bsub < testit.bash

This will run 6 jobs, PARAM1 will have range 0-1, PARAM2 will have range 0-2.

I'm making some assumptions about your model and day parameters.

BSUB many matlab job to a cluster?

2 Answers2