How can I use qsub with Python from the command line?

Question

I just tried this command:

qsub python commandLineDouble.py 1 10 1

and received this error:

Unable to read script file because of error: error opening python: No such file or directory

I'm running this on a computing cluster that can usually qsub .csh files quite fine.

hum, did you try to run python alone? "error opening python" doesn't look nice here :-/ — Samuele Mattiuzzo, Dec 11 '12 at 14:57
I think `qsub` is looking for a script named `python` to run in a shell. I don't think that `qsub` will run arbitrary commands. — mgilson, Dec 11 '12 at 15:09
Running Python alone works totally fine on the original system. But I wonder if there has to be a separate instance of Python for qsub too? — InquilineKea, Dec 11 '12 at 15:10
Hmm maybe I could try a shell file that executes a command like "time python DoublePendulum.py 1 1 1"? I'm new to shell scripting though. — InquilineKea, Dec 11 '12 at 15:10
I'd just first `qsub` a script where the only command is `which python`, and take it from there (e.g. paths, other environment variables etc) — ev-br, Dec 11 '12 at 15:16
Hm - I just tried that. I also tried qsub on a "hello world" script. It didn't display anything though. Hmm. — InquilineKea, Dec 11 '12 at 17:56

score 7 · Answer 1 · answered Mar 16 '13 at 16:33

7

use -S option

$ qsub -S $(which python) commandLineDouble.py 1 10 1

answered Mar 16 '13 at 16:33

mimori

71
2
3

score 6 · Answer 2 · answered Dec 11 '12 at 15:00

6

My impression is that qsub behaves differently from cluster to cluster. However in my experience, I've always just done:

qsub myscrip.sh

Note that this is different than:

qsub bash myscript.sh
qsub csh myscript.csh

In other words, qsub is a wapper around a shell (usually a particular shell, but since shell's are similar enough, this isn't usually an issue), not around arbitrary commands. The easy fix here is to just put your commands in a shell script and submit that:

#qsub directives here
#wrapper.sh
python commandLineDouble.py 1 10 1

Now you just do qsub wrapper.sh and Bob's your uncle (Assuming that your cluster's parallel environment has python installed...).

answered Dec 11 '12 at 15:00

mgilson

300,191
65
633
696

Wow - thanks so much! I have a class to go to now but I'll really try that out and see how that goes! – InquilineKea Dec 11 '12 at 15:11
1

@InquilineKea I think this will work (at least this used to work in my qsub processes). Please accept the answer if it works. – pranshus Dec 11 '12 at 15:14
Hmm - sadly - it doesn't work yet... Zhenya suggested a qsub on "which python" but that didn't display anything.. – InquilineKea Dec 11 '12 at 17:57
1

@InquilineKea -- Usually the environment which gets loaded in `qsub` is different than the environment you actually are working with. In your login terminal (outside your `qsub` script), try `which python`. You should get a path as you said that you can run the code outside of `qsub`. Then copy that entire path into your `qsub` script. e.g. if you see `/usr/local/bin/python` when you type `which python`, try putting `/usr/local/bin/python yourscript.py` into your qsub file. If that still doesn't work, I'd suggest you contact the system administrators. You might need `module load python` – mgilson Dec 11 '12 at 18:46

score 1 · Answer 3 · answered Aug 28 '13 at 23:55

1

On PBS Pro you need to use a double hyphen if you want to supply an actual command line instead of a script:

qsub -- python commandLineDouble.py 1 10 1

answered Aug 28 '13 at 23:55

jjellis

33
4

score 0 · Answer 4 · edited May 11 '13 at 23:55

0

Have you tried echo "script.py <args>" | qsub?

edited May 11 '13 at 23:55

Cairnarvon

25,981
9
51
65

answered May 11 '13 at 23:17

user2373938

1

How can I use qsub with Python from the command line?

4 Answers4

Linked