How to properly pass on an environment variable to Sun Grid Engine?

Question

I'm trying to submit a (series of) jobs to SGE (FWIW, it's a sequence of Gromacs molecular dynamics simulations), in which all the jobs are identical except for a suffix, such as input01, input02, etc. I wrote the commands to run in a way that the suffix is properly handled by the sequence of commands.

However, I can't find a way to get the exec environment to receive that variable. According to the qsub man page, -v var should do it.

$ export i=19
$ export | grep ' i='
declare -x i="19"
$ env | grep '^i='
i=19

Then, I submit the following script (run.sh) to see if it's received:

if [ "x" == "x$i" ]; then
    echo "ERROR: \$i not set"
else
    echo "SUCCESS: \$i is set"
fi

I submit the job as follows (in the same session as the export command above):

$ qsub -N "test_env" -cwd -v i run.sh
Your job 4606 ("test_env") has been submitted

The error stream is empty, and the output stream has:

$ cat test_env.o4606 
ERROR: $i not set

I also tried the following commands, unsuccessfully:

$ qsub -N "test_env" -cwd -v i -V run.sh
$ qsub -N "test_env" -cwd -V run.sh
$ qsub -N "test_env" -cwd -v i=19 -V run.sh
$ qsub -N "test_env" -cwd -v i=19 run.sh

If I add a line i=19 to the beginning of run.sh, then the output is:

$ cat test_env.o4613
SUCCESS: $i is set as 19

I'm now considering generating a single file per job, which will essentially be the same but will have an i=xx line as the first. It doesn't look very much practical, but it would be a solution.

Would there be a better solution?

`qsub -N "test_env" -cwd -v="$i" run.sh` should work just fine. — Simon O'Hanlon, Aug 13 '15 at 18:31

score 7 · Answer 1 · answered Jan 23 '13 at 13:30

7

What I've been always doing is the following:

##send.sh
export a=10
qsub ./run.sh

and the script run.sh:

##run.sh
#$ -V
echo $a

when I call send.sh, the .o has an output of 10.

answered Jan 23 '13 at 13:30

Jorge Leitao

19,085
19
85
121

Thanks. This involves the least extra typing, and does work even when the .sh script then calls another program (tested with R). – Ari B. Friedman May 04 '14 at 12:58

score 1 · Answer 2 · answered Feb 21 '13 at 22:19

I'm not certain you can pass variables by their name through qsub. I've had success with passing values (you should probably write a front-end script for this instead of doing it interactively):

$ export ii=19
$ qsub -N "test_env" -cwd -v i=$ii run.sh

score 1 · Answer 3 · answered Mar 15 '12 at 16:18

Assuming that your variable is just an incrementing counter: You can use Array Jobs to achieve this. This will set an $SGE_TASK_ID environment variable to the count which you can then copy to $i or use directly.

If the variable is anything else, then I think you'll have to generate multiple job scripts and submit each; that's the "solution" I use when I have loads of jobs with differing parameters.

How to properly pass on an environment variable to Sun Grid Engine?

3 Answers3