Is there a way to use qsub and source together?

Question

I wrote a shell script to process a bunch of files separately, like this

#!/bin/bash

#set parameters (I have many)
...

#find the files and do iteratively
for File in $FileList; do
    source MyProcessing.sh
done

MyProcessing.sh is the calling script, and the variables and functions from the main script are used in the calling script.

Now I'd like to move my shell script to cluster, and use qsub in the iteration. I tried

#find the files and do iteratively
for File in $FileList; do
    echo "source MyProcessing.sh" | qsub
done

But it does not work in this way. Anyone can help? Thank you in advance.

Answer 1

Variables and functions are local to a script. This means source MyProcessing.sh will work but bash MyProcessing.sh won't. The second syntax creates a subshell which means a new Unix process and Unix processes are isolated.

The same is true for qsub since you invoke it via a pipe: BASH will create a new process qsub and set the stdin to source MyProcessing.sh. That only passes these 23 bytes to qsub and nothing else.

If you want this to work, then you will have to write a new script that is 100% independent of the main script (i.e. it must not use any variables or functions). Then you must read the documentation of qsub to find out how to set it up. Usually, tools like that only work after you distributed a copy of MyProcessing.sh on every node of the cluster.

Also, the tool probably won't try to figure out what other data the script needs, so you will have to copy the files to the cluster nodes as well (probably by putting them on a shared file system).

Answer 2

Use:

(set; echo "source MyProcessing.sh") | qsub

You need to set your current variables in qsub shell.