How can I use bsub to consider two sets of inputs for my code

Question

I use the following shell script to send jobs to the server:

#!/bin/bash
#BSUB -J randJobName_ul0vm[1-3]%22
#BSUB -oo <pathToLog>/log/randJobName_ul0vm.o.%J.%I    
#BSUB -q "server.q"
#BSUB -n 2
#BSUB -R "span[hosts=1]"

JOB_ID=$LSB_JOBID
SGE_TASK_ID=$LSB_JOBINDEX
JOB_NAME=$LSB_JOBNAME
Id=$((SGE_TASK_ID-1))
declare -a Input
Input[0]="input1 input2"
Input[1]="input3 input4"
Input[2]="input5 input6"

python pythoncode.py parameter1 ${Input[${Id}]}

So, it gives the following inputs to the pthoncode.py:

parameter1 input1 input2
parameter1 input3 input4
parameter1 input5 input6

How can I modify it so that it also gives the following inputs?

parameter2 input1 input2
parameter2 input3 input4
parameter2 input5 input6

Note that parameter1 is replaced with parameter2 here.

I used the following code but it says index out of range. I don't understand why:

#!/bin/bash
#BSUB -J randJobName_ul0vm[1-6]%22
#BSUB -oo <pathToLog>/log/randJobName_ul0vm.o.%J.%I    
#BSUB -q "server.q"
#BSUB -n 2
#BSUB -R "span[hosts=1]"

JOB_ID=$LSB_JOBID
SGE_TASK_ID=$LSB_JOBINDEX
JOB_NAME=$LSB_JOBNAME
Id=$((SGE_TASK_ID-1))
declare -a Input
Input[0]="input1 input2"
Input[1]="input3 input4"
Input[2]="input5 input6"

python pythoncode.py parameter1 ${Input[${Id}]}
python pythoncode.py parameter2 ${Input[${Id}]}

Print `${Input[${Id}]}` every time you are about to use it to ensure your indexes are valid and the values are what you expect. For your "index out of range", what line returns that? Should we assume it is the last one? — Nic3500, Oct 11 '19 at 01:16

score 0 · Answer 1 · answered Oct 11 '19 at 14:19

This is your problem:

#BSUB -J randJobName_ul0vm[1-6]%22

In the first script, you're using an array of 3 elements so Id ranges from 0-2 for which you have Input defined. In the second script you're using an array of 6 elements so Id ranges from 0-5, and the three instances of the array using Id=3,4,5 will fail with the stated error because Input is not defined on those indices.

You need to change the above line back to

#BSUB -J randJobName_ul0vm[1-3]%22

in which case you will have 3 array elements, each one will run pythoncode.py with parameter1 and parameter2 sequentially. If you want these to execute in parallel (i.e. one array task to one execution of pythoncode.py and thus an array of 6 subjobs) you'll need to modify your script to something like this:

#!/bin/bash
#BSUB -J randJobName_ul0vm[1-6]%22
#BSUB -oo <pathToLog>/log/randJobName_ul0vm.o.%J.%I    
#BSUB -q "server.q"
#BSUB -n 2
#BSUB -R "span[hosts=1]"

JOB_ID=$LSB_JOBID
SGE_TASK_ID=$LSB_JOBINDEX
JOB_NAME=$LSB_JOBNAME

#### Squirrel changes begin #####

INDEX=$((SGE_TASK_ID-1))     # this ranges over 0,1,2,3,4,5 over the 6 tasks
InputIdx=$(( INDEX / 2 ))    # this ranges over 0,0,1,1,2,2
ParameterIdx=$(( INDEX / 3)) # this ranges over 0,0,0,1,1,1

declare -a Input
Input[0]="input1 input2"
Input[1]="input3 input4"
Input[2]="input5 input6"

declare -a Parameter
Parameter[0]="parameter1"
Parameter[1]="parameter2"

python pythoncode.py ${Parameter[${ParameterIdx}]} ${Input[${InputIdx}]}

This is wrong. It only uses paramter[0]. How can we fix this? — Albert, Nov 01 '19 at 21:11
Let me clarify. It uses paramter[0] Input[0] 6 times. How can we fix this? — Albert, Nov 01 '19 at 21:27

Ivan · Answer 2 · 2020-02-17T07:47:16.070

-1

According to this randJobName why don't you just use RANDOM?

parameter="parameter$((1+RANDOM%2))"

And in input

input=${Input[$((RANDOM%2))]}

edited Feb 17 '20 at 07:47

answered Feb 17 '20 at 06:53

Ivan

6,188
1
16
23

How can I use bsub to consider two sets of inputs for my code

2 Answers2