Popen.communicate is stuck until process spawned by sub-process terminates

Question

I have the below three scripts, and when I run main.py file, it spawns child.py which again executes the subchild.py and terminates quicly, subchild.py however keeps executing for a lot of time.

The problem with this is, main.py is blocked at p.communicate() till subchild.py terminates. If I open task manager and kill the running subchild.py main.py immediately returns the output of child.py

So my questions are as below

main.py is supposed to wait only till child.py terminates, why is it waiting for subchild.py to terminate ?
How do I make the p.communicate() not wait till subchild.py completes its execution ?

    # main.py file

    if __name__ == '__main__':
        import subprocess
        p = subprocess.Popen(
            ["python", "child.py"],
            stdout=subprocess.PIPE,
            stdin=subprocess.PIPE,
            stderr=subprocess.PIPE
        )
        out, _ = p.communicate()
        print(out)


    # child.py file

    if __name__ == '__main__':
        import subprocess
        print("Child running")
        p = subprocess.Popen(["python", "subchild.py"])
        print("Child pid - ", p.pid)
        exit(1)


    # subchild.py file

    if __name__ == '__main__':
        import time
        time.sleep(10000)

Note: I'm trying this on Windows 7 Enterprise. I'm using python3.6.6

Update after comment: On the main.py I need child.py's pid, stdout, stderr and process object so that I can kill child.py from main.py at any later point if I want to. This code is a small snippet is part of some API which I am building where the user would want the control to kill the process if he wishes to. subprocesses.call or subprocesses.run would not let me get control over the process object. I also won't have control over what child.py command I will receive as input for main.py, So I need to somehow not wait for the subchild.py and exit immediately with child.py's output as soon as it completes.

I might be missing something, but its waiting for subchild.py to finish because child.py can ONLY finish if subchild.py finishes. — LeKhan9, Oct 18 '18 at 21:26
no, child.py finishes before subchild.py finishes. After running the program you will only see main.py's PID and subchild.py's PID, child.py would have completed its execution. — vikkyhacks, Oct 18 '18 at 21:40
It think parent is waiting for stdout. If you run it without the `std*` arguments it will return the child's pid instantly. — solarc, Oct 18 '18 at 22:27

score 2 · Answer 1 · answered Oct 18 '18 at 21:52

Your communicate call doesn't just wait for the child to complete. It first tries to read the entire contents of the child's stdout and stderr.

Even when the child has completed, the parent can't stop reading, because the grandchild inherits the child's stdin, stdout, and stderr. The parent needs to wait for the grandchild to complete, or at least for the grandchild to close its stdout and stderr, before the parent can be sure it's done reading.

LeKhan9 · Answer 2 · 2018-10-18T22:04:07.493

Right now, child.py will always wait for subchild.py to finish. If you don't need to wait, you can try subprocess.call() as an alternative in the child.py code. Replace the Popen() logic with the call() logic instead.

p = subprocess.call(["python", "subchild.py"])

Here is a reference: https://docs.python.org/2/library/subprocess.html#subprocess.call

An alternative is to keep the code as it is an actually change to using .call() in the main.py file.

import subprocess
p = subprocess.call(
    ["python", "child.py"],
    stdout=subprocess.PIPE,
    stdin=subprocess.PIPE,
    stderr=subprocess.PIPE
)

You mentioned you can't use .call() because you'll need to kill it, however this is not the case. Making this change should allow you to:

return main.py and child.py quickly,
it will print the ID of the subchild process from child.py and
subchild.py will continue running

updated regarding your concern. I believe that might help – LeKhan9 Oct 18 '18 at 21:57 — LeKhan9, Oct 18 '18 at 21:57

score 1 · Accepted Answer · answered Oct 19 '18 at 06:52

1

Finally I found the solution to this myself, this seems to be an open issue with python. Would be glad to see this fixed. https://bugs.python.org/issue26731

answered Oct 19 '18 at 06:52

vikkyhacks

3,190
9
32
47

Popen.communicate is stuck until process spawned by sub-process terminates

3 Answers3