Python: How to read stdout non blocking from another process?

Question

During the runtime of a process I would like to read its stdout and write it to a file. Any attempt of mine however failed because no matter what I tried as soon as I tried reading from the stdout it blocked until the process finished.

Here is a snippet of what I am trying to do. (The first part is simply a python script that writes something to stdout.)

import subprocess

p = subprocess.Popen('python -c \'\
from time import sleep\n\
for i in range(3):\n\
    sleep(1)\n\
    print "Hello", i\
\'', shell = True, stdout = subprocess.PIPE)

while p.poll() == None:
    #read the stdout continuously
    pass

print "Done"

I know that there are multiple questions out there that deal with the same subject. However, none of the ones I found was able to answer my question.

@Keith Well, this is kind of my question. How do you read the stream without blocking it. I hope to find that part of the code in an answer somewhere below. ^^ — Woltan, Jun 20 '11 at 15:50

score 12 · Accepted Answer · answered Jun 20 '11 at 16:07

What is happening is buffering on the writer side. Since you are writing such small chunks from the little code snippet the underlying FILE object is buffering the output until the end. The following works as you expect.

#!/usr/bin/python

import sys
import subprocess

p = subprocess.Popen("""python -c '
from time import sleep ; import sys
for i in range(3):
    sleep(1)
    print "Hello", i
    sys.stdout.flush()
'""", shell = True, stdout = subprocess.PIPE)

while True:
    inline = p.stdout.readline()
    if not inline:
        break
    sys.stdout.write(inline)
    sys.stdout.flush()

print "Done"

However, you may not be expecting the right thing. The buffering is there to reduce the number of system calls in order to make the system more efficient. Does it really matter to you that the whole text is buffered until the end before you write it to a file? Don't you still get all the output in the file?

Hi keith, what if I wanted to send some commands to my process like `p.stdin.write(command1)`, then read the chunk of output from that (as in the output resulting only from this input!) and then, `p.stdin.write(command1)` and read that chunk and so on? — Dipole, Feb 24 '16 at 12:08
@Jack Yes, you can do that. You should use shell=False in that case, and stdin=PIPE. You can also use the `communicate` method to make that easier. See the [docs](https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate). — Keith, Feb 24 '16 at 19:17

Corey Goldberg · Answer 2 · 2011-06-21T13:34:17.700

5

the following code would print stdout line by line as the subprocess runs until the readline() method returns an empty string:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, ''):
    print line
p.stdout.close()
print 'Done'

update relating to your question better:

import subprocess

p = subprocess.Popen(['python'], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
p.stdin.write("""
from time import sleep ; import sys
for i in range(3):
    sleep(1)
    print "Hello", i
    sys.stdout.flush()
""")
p.stdin.close()
for line in iter(p.stdout.readline, ''):
    print line
p.stdout.close()
print 'Done'

edited Jun 21 '11 at 13:34

answered Jun 20 '11 at 15:59

Corey Goldberg

59,062
28
129
143

Removing the parameter `shell = True` from the `Popen` constructor results in a `OSError: [Errno 2] No such file or directory` exception on my machine. And if I leave the `shell = True` parameter in there, your reading routine still blocks the reading process. – Woltan Jun 21 '11 at 06:16
I pass it exactly like posted in my question. I simply removed the `shell = True` statement. – Woltan Jun 21 '11 at 12:55

score -3 · Answer 3 · answered Jun 20 '11 at 15:41

-3

You can use subprocess.communicate() to get the output from stdout. Something like:

while(p.poll() == None):
    #read the stdout continuously
    print(p.communicate()[0])
    pass

More info available at: http://docs.python.org/library/subprocess.html

answered Jun 20 '11 at 15:41

iceaway

1,204
1
8
12

1

Unfortunately the `communicate` method also blocks and only returns the stdout after the process `p` has finished. No continuous readout is possible. – Woltan Jun 20 '11 at 15:43

Python: How to read stdout non blocking from another process?

3 Answers3