Submitting jobs with numeric arguments to QSUB using Python

Question

I want to run a python function,say my_fun(x1,x2), on different nodes of cluster with SGE (QSUB). I created a script my_script.py which takes in the numeric arguments from the command line, so when running locally, I'd call it as

python my_script.py x1 x2

Now I want to submit this script to the cluster in a loop with different values of x1 and x2. Also, for the node to have access to python & the installed modules, I need to run module load Python/2.7 on the node before calling the python script through QSUB.

This seems like a pretty simple and typical use-case, but I can't find any straightforward way to do this from Python. Going back and forth between BASH and Python seems a bit clunky.

please, have a look at array jobs – Dima Chubarov Jun 06 '16 at 06:04 — Dima Chubarov, Jun 06 '16 at 06:04

Ben K. · Answer 1 · 2016-06-03T16:29:55.150

1

I suggest you divide the job into multiple independent jobs depending on the number of nodes you have.

For each node/core, create a folder with a file containing the a list of the parameters this subjob should treat. Then in python, write a script which reads the file and calls your script (maybe using the multiprocessing module for multi-core support).

EDIT:

If you want to pass additional parameters through qsub, you can call qsub with arguments which can be passed to your script:

qsub -F "myarg1 myarg2 myarg3=myarg3value" myscript.sh

You can find this documentation here

edited Jun 03 '16 at 16:29

answered Jun 03 '16 at 16:04

Ben K.

1,160
6
20

That's horribly inefficient for the number of jobs I need to submit. I think there are ways to do this without requiring writing files to disk. – KartMan Jun 03 '16 at 16:06
where do your x1 x2 values come from? because qsub is also inefficient for calling small scripts – Ben K. Jun 03 '16 at 16:08
Another suggestion if you are looking for a way to farm out your jobs would be to use ipython/jupyter which allows you to start up a bunch of workers, submit calculations and collect the results. – Ben K. Jun 03 '16 at 18:34
That would have been my preferred way actually, but unfortunately, the ipengines kept crashing on importing one module (sklearn.svm). Doing the import through QSUB seems to be fine. I have no idea why. Here's my post on that issue : http://stackoverflow.com/questions/37531089/ipython-parallel-on-sge-cluster-scikit-learn-svc-parallel-import-causes-engine – KartMan Jun 03 '16 at 18:48
That is weird. Sorry I can't help you there. – Ben K. Jun 03 '16 at 19:21

score 0 · Accepted Answer · answered Jun 03 '16 at 16:43

This more or less does what I'm looking for:

https://gist.github.com/timflutre/a9085660271bd059f71c

import sys
import subprocess

job_param1 = 12.5
job_param2 = 5.0
jobName = "python my_script.py %f %f" % (job_param1,job_param2)
cmd = "module load Python/2.7; sleep 0.2; %s" % jobName
echoArgs = ["echo", "-e", "'%s'" % cmd]
print(" ".join(echoArgs))
qsubArgs = ["qsub","-cwd"]
print(" ".join(qsubArgs))

wholeCmd = " ".join(echoArgs) + " | " + " ".join(qsubArgs)
out = subprocess.Popen(wholeCmd, shell=True, stdout=subprocess.PIPE)
out = out.communicate()[0]

jobId = out.split()[2]
print jobId

Of course, I'll be running the body inside a (python) FOR loop. — KartMan, Jun 03 '16 at 16:44

Submitting jobs with numeric arguments to QSUB using Python

2 Answers2