3

I would like to submit qsub jobs on the fly without creating discrete job files. So, let's say I have a python script called "get_time.py" that simply reports the time. Instead of making a submission script like this:

cat>job.sub<<eof
    #PBS -l walltime=1:00:00
    cd $PBS_O_WORKDIR
    get_time.py
eof

...and then submitting the job: qsub job.sub

I would like to be able to bypass the file creation step, and I'd image the construct would be something like this: qsub -d . -e get_time.py

where -e is my imaginary parameter that tells qsub that the following is code to be sent to the scheduler, instead of using a discrete submission file.

Am I crazy? It seems like there should already be a simple solution for this, but I couldn't find it anywhere. Thanks for any suggestions!

jake9115
  • 3,964
  • 12
  • 49
  • 78

6 Answers6

4

If you are ok with writing your job in a HERE document, as in your example, you can pass that HERE doc directly to qsub without a temporary file:

qsub <<eof
    #PBS -l walltime=1:00:00
    cd \$PBS_O_WORKDIR
    get_time.py
eof

Also note that you need to escape the dollar sign in $PBS_O_WORKDIR or else it will be interpolated before submitting the job and end up as an empty string.

Ryan C. Thompson
  • 40,856
  • 28
  • 97
  • 159
  • any idea how to pass script arguments this way as well? If I do `qsub $arg1 < – user5359531 Jul 18 '16 at 17:24
  • You can put any code you like into the HERE document. Just put arg1 after get_time.py. – Ryan C. Thompson Jul 18 '16 at 17:55
  • Right but that does not work for qsub specific items like `$NSLOTS` which do no exist until after qsub has run; it gets evaluated to an empty string by the heredoc. However I got around this by escaping it within the heredoc; `\$NSLOTS`. You can also escape the file string to prevent all expansion e.g. `<<"EOF"`. http://tldp.org/LDP/abs/html/here-docs.html – user5359531 Jul 18 '16 at 18:24
  • Yes, that's correct, any variables inside the heredoc will be interpolated before submission, not after, unless you escape them as you have described. (Actually, I should edit my answer to reflect that.) – Ryan C. Thompson Jul 18 '16 at 18:26
3

You can pass the name of any executable script/file to qsub after you've provided all your options (-d ., etc.). Any arguments that come after the script name are treated as arguments for that script. For example, if I had the script test.py:

#!/usr/bin/python
from sys import argv
script, param = argv
print param

Then I can run

qsub test.py 2

and the output to my log will be 2.

Two important things to note:

  1. Your Python script must be executable (e.g. chmod +x test.py).
  2. Your Python script must have a shebang (e.g. #!/usr/bin/python).
mdml
  • 22,442
  • 8
  • 58
  • 66
  • 1
    This is what I thought. However, I have a python script which has permissions -rwxr-xr-x and the shebang line `#!/usr/bin/env python2.7` and qsub fails for `qsub -N test_script -o logs -e logs -V -cwd /path/to/script.py`. How do I fix this? – Arjun Sep 10 '15 at 20:47
  • yeah, doesn't work for me either on OGS/GE 2011.11p1 – cbare Sep 11 '15 at 06:54
3

In grid engine, -b y specifies that the thing to be executed is a binary, not a script:

qsub -b y get_time.py
Sujai
  • 33
  • 2
  • This is true only for the Sun Grid Engine I believe. The `-b` option means something else in TOURQUE clusters. See the qsub man page for more details [link](http://linux.die.net/man/1/qsub-torque) – macKaiver Jun 07 '16 at 08:37
1

I don't understand the question but...

echo 'date' | qsub

is not working well for you?

deggio
  • 31
  • 3
0

Provided get_time.py is executable, and the shebang is correct, you should be able to simply run

qsub -d . get_time.py

If that fails for either reason, you can use the following construct

qsub -d . <<< "python get_time.py"
damienfrancois
  • 52,978
  • 9
  • 96
  • 110
0

The easiest for me is to add the command(s) I want to run in an echo-statement and pipe it to qsub. For example, in this case:

echo "get_time.py" | qsub -d . -l walltime=1:00:00
Keivan
  • 1,300
  • 1
  • 16
  • 29
ashotmarg2004
  • 1