Making qsub block until job is done?

Question

Currently, I have a driver program that runs several thousand instances of a "payload" program and does some post-processing of the output. The driver currently calls the payload program directly, using a shell() function, from multiple threads. The shell() function executes a command in the current working directory, blocks until the command is finished running, and returns the data that was sent to stdout by the command. This works well on a single multicore machine. I want to modify the driver to submit qsub jobs to a large compute cluster instead, for more parallelism.

Is there a way to make the qsub command output its results to stdout instead of a file and block until the job is finished? Basically, I want it to act as much like "normal" execution of a command as possible, so that I can parallelize to the cluster with as little modification of my driver program as possible.

Edit: I thought all the grid engines were pretty much standardized. If they're not and it matters, I'm using Torque.

score 4 · Answer 1 · answered May 12 '11 at 20:19

4

You don't mention what queuing system you're using, but SGE supports the '-sync y' option to qsub which will cause it to block until the job completes or exits.

answered May 12 '11 at 20:19

Steve Baker

4,323
1
20
15

score 2 · Answer 2 · answered Mar 13 '12 at 22:36

2

In TORQUE this is done using the -x and -I options. qsub -I specifies that it should be interactive and -x says run only the command specified. For example:

qsub -I -x myscript.sh

will not return until myscript.sh finishes execution.

answered Mar 13 '12 at 22:36

dbeer

6,963
3
31
47

Don't you lose the ability to store the output in output files? – static_rtti Jul 24 '12 at 08:27
1

You lost the automatic storing of output this way, but you can log your terminal session and save the output that way. – dbeer Jul 24 '12 at 15:13

score 1 · Answer 3 · answered Sep 16 '16 at 06:15

1

In PBS you can use qsub -Wblock=true <command>

answered Sep 16 '16 at 06:15

naught101

18,687
19
90
138

Making qsub block until job is done?

3 Answers3