I am submitting a job(script) to Sun Grid Engine. The job is a python program. It can take many hours to run, but it will periodically write to stdout and stderr to inform me its status (like how many iterations is finished, etc). The problem is that SGE is buffering the output and only writes to file at the end, which means that I cannot see the output on the screen or by tailing the file in real time. I can only get to know the status after the job is finished. Is there a way to get around this by configuring SGE (qsub, etc.)?
Asked
Active
Viewed 1,170 times
3
-
1Try using sys.stdout.write() and sys.stdout.flush() rather than print() – jgritty Oct 31 '14 at 04:09
1 Answers
0
Instead of using qsub, use qrsh.
Finch_Powers
- 2,938
- 1
- 24
- 34
-
How do I change my `qsub` to a `qrsh` command? Currently the command looks like: `qsub -V -b n -cwd -j y -S /usr/bin/python`. `qrsh` seems to support a different set of commands. – Alex Rothberg Jan 06 '15 at 20:57
-
I haven't tried directly with interractive calls such as python. For non interractive calls it's straight forward. Eg.: "qrsh -V -cwd -j y pwd && hostname". For interractive calls you can do it in two steps. First call "qrsh" wich will log you to a worker node. Then call "python". – Finch_Powers Jan 06 '15 at 22:22