Python Subprocess Wait for any subprocess which finishes first

Question

I'm currently running some subprocesses in parallel(multiple subprocesses), {p1, p2, p3, p4}.

I'd like to wait() until any of them finishes.

I'm currently polling in a while loop which is probably very inefficient

proc = [p1, p2, p3, p4]
while True:
  for p in proc:
    if p.poll() != None:
      #Do whatever

I was wondering, is there a way to wait for the fastest finishing subprocess instead of busy waiting polling through all the subprocesses?

score 1 · Answer 1 · answered Mar 31 '21 at 08:16

So long as you're not on Windows, you can use os.wait() for this. It is exactly designed to wait until the first child process exits.

However, a hidden side effect is that you lose the process's exit code (it will now be presumed to be 0). It is possible to set it yourself, but it's kind of hacky.

proc = [p1, p2, p3, p4]
pid, status = os.wait()
for p in proc:
    if p.pid == pid:
        # We need to set the process's exit status now, or we
        # won't be able to retrieve it later and it will be
        # assumed to be 0.
        # This is a kind of hacky solution, but this function has existed
        # ever since subprocess was first included in the stdlib and is
        # still there in 3.10+, so it *should* be pretty stable.
        p._handle_exitstatus(status)

        #Do whatever

Note: this all works equally well on python 3

Python Subprocess Wait for any subprocess which finishes first

1 Answers1