Forward subprocess stdout does not work when stdout is redirected to a file

Question

I have a problem forwarding the stdout of a subprocess to stdout of the current process.

This is my MWE caller code (runner.py):

import sys
import subprocess
import time

p = subprocess.Popen([sys.executable, "test.py"], stdout=sys.stdout)
time.sleep(10)
p.terminate()

and here is the content of the callee test.py:

import time

while True:
    time.sleep(1)
    print "Heartbeat"

The following will work and print all the heartbeats to the console:

python runner.py

However, the following does not work, the output text file remains empty (using Python 2.7):

python runner.py > test.txt

What do I have to do?

@zondo No, I do not want to have the shell commands included — Georg, Mar 04 '16 at 16:41

Andrea Corbellini · Accepted Answer · 2016-03-04T16:30:30.520

7

When the standard output is a TTY (a terminal), sys.stdout is line-buffered by default: each line you print is immediately written to the TTY.

But when the standard output is a file, then sys.stdout is block-buffered: data is written to the file only when a certain amount of data gets printed. By using p.terminate(), you are killing the process before the buffer is flushed.

Use sys.stdout.flush() after your print and you'll be fine:

import sys
import time

while True:
    time.sleep(1)
    print "Heartbeat"
    sys.stdout.flush()

If you were using Python 3, you could also use the flush argument of the print function, like this:

import time

while True:
    time.sleep(1)
    print("Heartbeat", flush=True)

Alternatively, you can also set up a handler for SIGTERM to ensure that the buffer is flushed when p.terminate() gets called:

import signal
signal.signal(signal.SIGTERM, sys.stdout.flush)

edited Mar 04 '16 at 16:30

answered Mar 04 '16 at 16:18

Andrea Corbellini

17,339
3
53
69

1

1- it is error-prone (and inefficient) to call `sys.stdout.flush()` manually all over your program. You can unbuffer stdout for the whole process instead 2- `signal(signal.SIGTERM, sys.stdout.flush)` won't work. You might mean `signal(signal.SIGTERM, lamba *a: (sys.stdout.flush(), sys.exit(-signal.SIGTERM)))` instead. Though it might be simpler, to [call `p.send_signal(signal.SIGINT)` instead of `p.terminate()` in the parent](http://stackoverflow.com/a/35819206/4279). – jfs Mar 05 '16 at 20:10
4

@J.F.Sebastian: flushing output is actually *the* way. You don't have to trust me: look at `ping`, `tail` or every other tool in your system that writes output periodically – Andrea Corbellini Mar 06 '16 at 08:25
Why do you think `grep` has `--line-buffered` parameter? Why doesn't it flush after each line by default as you're suggesting? – jfs Mar 06 '16 at 12:06

Antti Haapala -- Слава Україні · Answer 2 · 2016-03-04T16:31:50.817

1

It is possible to force flushes by doing sys.stdout.flush() after each print, but this would quickly become cumbersome. Since you know you're running Python, it is possible to force Python to unbuffered mode - either with -u switch or PYTHONUNBUFFERED environment variable:

p = subprocess.Popen([sys.executable, '-u', 'test.py'], stdout=sys.stdout)

or

import os
# force all future python processes to be unbuffered
os.environ['PYTHONUNBUFFERED'] = '1'

p = subprocess.Popen([sys.executable, 'test.py'])

edited Mar 04 '16 at 16:31

answered Mar 04 '16 at 16:26

Antti Haapala -- Слава Україні

129,958
22
279
321

you could pass `env=dict(os.environ, PYTHONUNBUFFERED=1)` to `Popen()` to avoid affecting other python processes. – jfs Mar 05 '16 at 19:29

score 1 · Answer 3 · edited May 23 '17 at 12:31

You don't need to pass stdout=sys.stdout unless sys.stdout uses a different file descriptor than the one that python executable has started with. C stdout fd is inherited by default: you don't need to do anything in order for the child process to inherit it.

As @Andrea Corbellini said, if the output is redirected to a file then python uses a block-buffering mode and "Heartbeat"*10 (usually) is too small to overflow the buffer.
I would expect that python flushes its internal stdout buffers on exit but it doesn't do it on SIGTERM signal (generated by the .terminate() call).
To allow the child process to exit gracefully, use SIGINT (Ctrl+C) instead of p.terminate():

p.send_signal(signal.SIGINT)

In that case test.py will flush the buffers and you'll see the output in test.txt file. Either discard stderr or catch KeyboardInterrupt exception in the child.

If you want to see the output while the child process is still running then run python -u, to disable buffering or set PYTHONUNBUFFERED envvar to change the behavior of all affected python processes as @Antti Haapala suggested.

Note: your parent process may also buffer the output. If you don't flush the buffer in time then the output printed before test.py is even started may appear after its output in the test.txt file. The buffers in the parent and the child processes are independent. Either flush buffers manually or make sure that an appropriate buffering mode is used in each process. See Disable output buffering

Forward subprocess stdout does not work when stdout is redirected to a file

3 Answers3