Why is standard output from subprocess (redirected to unbuffered file) being buffered?

Question

From http://docs.python.org/library/functions.html#open

The optional bufsize argument specifies the file’s desired buffer size: 0 means unbuffered, 1 means line buffered, any other positive value means use a buffer of (approximately) that size. A negative bufsize means to use the system default, which is usually line buffered for tty devices and fully buffered for other files. If omitted, the system default is used.

I'm passing 0 as bufsize below yet without using flush() there's no output written to the file when I run main_process.
What's the reason?

# --------------------------------- sub_process.py
import sys
import time

if __name__ == '__main__':
    print 'printed from redirect.py'
    # why is the following flush() needed? 'std-output' is (?) unbuffered...
    sys.stdout.flush() 
    time.sleep(6)


# --------------------------------- main_process.py
import subprocess
import time

if __name__ == '__main__':
    p = subprocess.Popen(
        ['python', 'sub_process.py'],
        stdout=open('std-output', 'w', 0))
    time.sleep(3)
    p.terminate()

+1, I spent about 30 minutes trying to figure out why `sys.stdout` -> `subprocess.PIPE` wasn't working a couple of days ago. `flush()` is the answer, but why do we need it??? — Mike Pennington, May 07 '11 at 17:21

score 8 · Answer 1 · answered May 07 '11 at 16:53

8

Use python with the -u flag, e.g.:

if __name__ == '__main__':
    p = subprocess.Popen(
        ['python', '-u', 'sub_process.py'],
        stdout=open('std-output', 'w'))
    time.sleep(3)
    p.terminate()

answered May 07 '11 at 16:53

rtn

127,556
20
111
121

This works, thanks. Though, I'm still wondering why it's needed at all? – Piotr Dobrogost May 07 '11 at 17:15
1

My guess is that if you write directly to the file with file.write() it will be done unbuffered, but you are in fact writing indirectly to it since python will buffer your print commands before sending the data to your file, hence you need -u flag to tell python to not buffer the print commands. – rtn May 07 '11 at 17:25
`sys.stdout.write('printed from redirect.py')` gives the same result as print - nothing gets written to a file – Piotr Dobrogost May 07 '11 at 17:45

mouad · Accepted Answer · 2011-05-07T20:00:28.447

Extending Magnus Skog solution (+1 by the way :) ):

Well basically what happen is that when subprocess will fork a new process it will duplicate the stdout argument to the new subprocess stdout (fileno = 1) using os.dup2 (look at subprocess.Popen._execute_child) and this will keep the unbuffered state (as what dup2 do), everything until now is good, but when python will be launched (in the subprocess) by default if python don't see the -u flag it will set the buffer of stdout to line buffer (take a look at the main python function.) which will override the buffering flag that you set before.

Hope this explain more the behavior that you was seeing.

Why is standard output from subprocess (redirected to unbuffered file) being buffered?

2 Answers2