0

Tried searching for the solution to this problem but due to there being a command Shell=True (don't think that is related to what I'm doing but I could well be wrong) it get's lots of hits that aren't seemingly useful.

Ok so the problem I is basically:

I'm running a Python script on a cluster. On the cluster the normal thing to do is to launch all codes/etc. via a shell script which is used to request the appropriate resources (maximum run time, nodes, processors per node, etc.) needed to run the job. This shell then calls the script and away it goes.

This isn't an issue, but the problem I have is my 'parent' code needs to wait for it's 'children' to run fully (and generate their data to be used by the parent) before continuing. This isn't a problem when I don't have the shell between it and the script but as it stands .communicate() and .wait() are 'satisfied' when the shell script is done. I need to to wait until the script(s) called by the shell are done.

I could botch it by putting a while loop in that needs certain files to exist before breaking, but this seems messy to me.

So my question is, is there a way I can get .communicate (idealy) or .wait or via some other (clean/nice) method to pause the parent code until the shell, and everything called by the shell, finishes running? Ideally (nearly essential tbh) is that this be done in the parent code alone.

I might not be explaining this very well so happy to provide more details if needed, and if somewhere else answers this I'm sorry, just point me thata way!

Steve
  • 614
  • 1
  • 10
  • 20
  • It sounds like you need the shell script that Python calls to not exit before the scripts it calls finishes. Is there a reason the shell script exits before the scripts it calls finishes? – OpenUserX03 Mar 06 '15 at 18:38
  • Not that I'm aware. I haven't been using the cluster long so not sure if keeping the shell from exiting would cause any issues. I guess it might be possible to edit the call function in the shell to wait. – Steve Mar 06 '15 at 18:39
  • When the shell script calls a program, it waits for that program to finish unless the script uses `&` to background it. Of course, the program itself may background more children and exit early. In either case, the script or the things that it runs should have mechanisms to handle backgrounded programs if they are to be at all robust. Maybe a pidfile or something. Otherwise, you've got some unpleasant work ahead of you because you are trying to run a script in a way its not designed to run. – tdelaney Mar 06 '15 at 19:07
  • As far as I can see there's no reason why the shell should exit. It's linux so I believe I'm right in saying that by default it shouldn't and you have to actively opt for it to be background? The shell seems to exit itself automatically, and I can't work out why. My hands are tied with a lot of the children (shell and script(s)) as they're either required by the cluster or are part of an external program. It's looking like I'll have to use the messy method to avoid this issue. – Steve Mar 06 '15 at 19:28

0 Answers0