Python os.dup2 redirect enables output buffering on windows python consoles

Question

I'm using a strategy based around os.dup2 (similar to examples on this site) to redirect C/fortran level output into a temporary file for capturing.

The only problem I've noticed is, if you use this code from an interactive shell in windows (either python.exe or ipython) it has the strange side effect of enabling output buffering in the console.

Before capture sys.stdout is some kind of file object that returns True for istty(). Typing print('hi') causes hi to be output directly. After capture sys.stdout points to exactly the same file object but print('hi') no longer shows anything until sys.stdout.flush() is called.

Below is a minimal example script "test.py"

import os, sys, tempfile

class Capture(object):
    def __init__(self):
        super(Capture, self).__init__()
        self._org = None    # Original stdout stream
        self._dup = None    # Original system stdout descriptor
        self._file = None   # Temporary file to write stdout to

    def start(self):
        self._org = sys.stdout
        sys.stdout = sys.__stdout__
        fdout = sys.stdout.fileno()
        self._file = tempfile.TemporaryFile()
        self._dup = None
        if fdout >= 0:
            self._dup = os.dup(fdout)
            os.dup2(self._file.fileno(), fdout)

    def stop(self):
        sys.stdout.flush()
        if self._dup is not None:
            os.dup2(self._dup, sys.stdout.fileno())
            os.close(self._dup)
        sys.stdout = self._org
        self._file.seek(0)
        out = self._file.readlines()
        self._file.close()
        return out

def run():
    c = Capture()
    c.start()
    os.system('echo 10')
    print('20')
    x = c.stop()
    print(x)

if __name__ == '__main__':
    run()

Opening a command prompt and running the script works fine. This produces the expected output:

python.exe test.py

Running it from a python shell does not:

python.exe
>>> import test.py
>>> test.run()
>>> print('hello?')

No output is shown until stdout is flushed:

>>> import sys
>>> sys.stdout.flush()

Does anybody have any idea what's going on?

---

Quick info:

• The issue appears on Windows, not on linux (so probably not on mac).
• Same behaviour in both Python 2.7.6 and Python 2.7.9
• The script should capture C/fortran output, not just python output
• It runs without errors on windows, but afterwards print() no longer flushes

Answer 1

I could confirm a related problem with Python 2 in Linux, but not with Python 3

The basic problem is

>>> sys.stdout is sys.__stdout__
True

Thus you are using the original sys.stdout object all the time. And when you do the first output, in Python 2 it executes the isatty() system call once for the underlying file, and stores the result.

You should open an altogether new file and replace sys.stdout with it.

---

Thus the proper way to write the Capture class would be

import sys
import tempfile
import time
import os

class Capture(object):
    def __init__(self):
        super(Capture, self).__init__()

    def start(self):
        self._old_stdout = sys.stdout
        self._stdout_fd = self._old_stdout.fileno()
        self._saved_stdout_fd = os.dup(self._stdout_fd)
        self._file = sys.stdout = tempfile.TemporaryFile(mode='w+t')
        os.dup2(self._file.fileno(), self._stdout_fd)

    def stop(self):
        os.dup2(self._saved_stdout_fd, self._stdout_fd)
        os.close(self._saved_stdout_fd)
        sys.stdout = self._old_stdout
        self._file.seek(0)
        out = self._file.readlines()
        self._file.close()
        return out

def run():
    c = Capture()
    c.start()
    os.system('echo 10')
    x = c.stop()
    print(x)
    time.sleep(1)
    print("finished")

run()

With this program, in both Python 2 and Python 3, the output will be:

['10\n']
finished

with the first line appearing on the terminal instantaneously, and the second after one second delay.

---

This would fail for code that import stdout from sys, however. Luckily not much code does that.