Python subprocess & stdout - program deadlocks

Question

I have a simulation program which is piloted though stdin and provides output to stdout

Doing a C++/Qt program for running it in a QProcess works well.

Doing a Python program for running it under linux works well, using:

p = subprocess.Popen(cmd,stdin=subprocess.PIPE,stdout=subprocess.PIPE)

And using p.stdin.write, p.stdout.readline, and p.wait

However, under windows, the program runs and gets the commands through stdin as it should(this has been verified by debugging the subprocess), but the python program deadlocks at any p.stdout.readline, and p.wait. If the stdout=subprocess.PIPE parameter is removed, the program works, the output is displayed on the console and no deadlock occurs.

This sounds familiar with a warning from the Python documentation:

Warning : This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.

However, I can't use communicate(), as the program protocol is not a single command and a single output, rather several commands and replies are required.

Is there any solution?

Answer 1

Unsure of it, but it looks like a buffering problem. On Linux (as on most Unix or Unix-like), output to a file or a pipe is internally buffered at the OS level. That means that after a write call, all the data is buffered but nothing is available at the other end of the pipe until either the internal buffer is full, the data is flushed or the pipe is closed. That's one of the reasons why ptys were invented and are not implemented with a pipe pair.

Said differently, it is not possible to drive a program where you need to use previous output to know what you should give as input with pipes, unless the program has been specially tailored for it by consistently flushing its output before reading anything. It works on a true terminal (tty or pty) because the driver automatically forces a flush of the output before any read on the same device.

But it is not the same dealock that is described in the documentation that you have cited in your question.