Use variables as argument parameters inside qsub script

Question

I want to pick up a number of models from a folder and use them in an sge script for an array job. So I do the following in the SGE script:

MODELS=/home/sahil/Codes/bistable/models
numModels=(`ls $MODELS|wc -l`)
echo $numModels 

#$ -S /bin/bash
#$ -cwd 
#$ -V
#$ -t 1-$[numModels] # Running array job over all files in the models directory.

model=(`ls $MODELS`)
echo "Starting ${model[$SGE_TASK_ID-1]}..."

But I get the following error:

Unable to read script file because of error: Numerical value invalid!
The initial portion of string "$numModels" contains no decimal number

I have also tried to use

#$ -t 1-${numModels}

and

#$ -t 1-(`$numModels`)

but none of these work. Any suggestions/alternate methods are welcome, but they must use the array job functionality of qsub.

score 3 · Accepted Answer · answered Sep 12 '14 at 13:04

3

Beware that to Bash, #$ -t 1-$[numModels] is nothing more than a comment; hence it does not apply variable expansion to numModels.

One option is to pass the -t argument in the command line: remove it from your script:

#$ -S /bin/bash
#$ -cwd 
#$ -V

model=(`ls $MODELS`)
echo "Starting ${model[$SGE_TASK_ID-1]}..."

and submit the script with

MODELS=/home/sahil/Codes/bistable/models qsub -t 1-$(ls $MODELS|wc -l) submit.sh

If you prefer to have a self-contained submission script, another option is to pass the content of the whole script through stdin like this:

#!/bin/bash
qsub <<EOT
MODELS=/home/sahil/Codes/bistable/models
numModels=(`ls $MODELS|wc -l`)
echo $numModels 

#$ -S /bin/bash
#$ -cwd 
#$ -V
#$ -t 1-$[numModels] # Running array job over all files in the models directory.

model=(`ls $MODELS`)
echo "Starting ${model[$SGE_TASK_ID-1]}..."
EOT

Then you source or execute that script directly to submit your job array (./submit.sh rather than qsub submit.sh as the qsub command is here part of the script.

answered Sep 12 '14 at 13:04

damienfrancois

52,978
9
96
110

This is valid, but I would really like if the solution is inside the sge script itself, not wrapped by a shell script, or by putting the arguments in the command line. – Sahil M Sep 12 '14 at 13:17
@Sahil, this is likely not possible. The SGE script will execute once per task, from 1 to the total task number calculated. So, determining the total number of tasks needs to happen before the SGE script is executed. This is what Damien is eluding to. – Vince Sep 19 '14 at 13:26
@Vince, I follow that, but -t anyways takes a static numeric argument in the SGE script. All I am saying is to replace the upper bound with a variable, it is not going to change with each task of the array job. If there is no provision for this, I'll probably just write the number by hand each time to avoid wrapping qsub by sh, which has its own set of problems. – Sahil M Sep 21 '14 at 13:36
You are correct. The only two solutions I see are type the max task id by hand or wrap it up in a bash script. – Vince Sep 22 '14 at 13:55

Use variables as argument parameters inside qsub script

1 Answers1