2

I have a subprocess that can output a lot of data to stdout. When too much data is generated it causes the subprocess to hang since it is waiting for the stdout buffer to be emptied.

Here is a small example below...

test.py

#!/usr/local/bin/python2.7

# test.py

import subprocess

proc = subprocess.Popen(["python","./_ping_con.py"], stdout = subp.PIPE)

while proc.poll() is None:
    pass

print proc.stdout.read()

...and the subprocess:

#!/usr/local/bin/python2.7

# _ping_con.py

print(96000 * "*")   # Hangs here because it's too much data for the stdout pipe

What I'd like to know is, can this buffer be expanded to allow for more data to be handled? If not, is there a different way I could be sending my data that would avoid this issue? OR in the main process, is there a way to tell if the stdout buffer is full and do a read?

Calab
  • 351
  • 4
  • 20
  • Related: https://stackoverflow.com/questions/2408650/why-does-python-subprocess-hang-after-proc-communicate – rdas Apr 17 '19 at 09:03
  • 3
    It's a known bug: " Note This will deadlock when using stdout=PIPE or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use Popen.communicate() when using pipes to avoid that. " – rdas Apr 17 '19 at 09:04
  • @rdas, _known_, yes. I'm not sure that's a _bug_ -- it's behaving as designed and documented. – Charles Duffy Aug 26 '20 at 19:16

1 Answers1

-2

What I ended up doing was writing all my output to a temporary file, then passing the name of the file back to the parent. Works very well!

Calab
  • 351
  • 4
  • 20
  • 1
    While this may work you fixed nothing really. Just masked the problem. You should fix it as per @rdas comment – Marcin Orlowski Apr 23 '19 at 03:19
  • How does using Popen.communicate() help if it closes the PIPE before I have finished using it? That's what I found when I looked online for simple examples of using it. My application has no requirement where I need to pass my data via the PIPE. – Calab Apr 23 '19 at 03:28