subprocess.Popen : Separating stdout/stderr but keeping order

Question

How do I get the output from a command using subprocess.Popen and have separate callbacks for stdout and stderr, but ensure that those callbacks are called in the order that the lines came from the process?

If I didn't care about separating out STDOUT and STDERR then I could do:

fd = subprocess.Popen( cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT )
line = fd.stdout.readline()
while line :
    callback( line )
    line = fd.stdout.readline()

However, if I have stdoutCallback and stderrCallback, and want them called on the appropriate outputs, but in the same order as the above code would call callback on, how would I go about doing this?

I'm thinking that I could do this myself by spawning a couple of threads, one for each of STDOUT and STDERR that would insert each line of output into a shared list, with an identifier as to which pipe the line came from. The main thread could then watch this list and call the appropriate callback. Mutex lock joy! — Hugh, Jan 12 '16 at 17:22

score 1 · Answer 1 · answered Jan 12 '16 at 16:32

1

fd = subprocess.Popen( cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE )
output,error = fd.communicate()

Use communicate

answered Jan 12 '16 at 16:32

vks

67,027
10
91
124

1

Thanks. This isn't quite what I'm after though. This appears to wait until the process is finished and then give me a tuple of the complete ( stdout, stderr ) tuple. I wrote a test little python script that prints "stdout 1" then "stderr 2", then "stdout 3" then "stderr 4", with 1 second delay between each. Using fd.communicate() gives me ("stdout 1\nstdout 3\n", "stderr 2\nstderr 4") with no indication what the relative order was. – Hugh Jan 12 '16 at 16:40
it doesn't preserve the order. – jfs Jan 13 '16 at 11:01

score 1 · Answer 2 · answered Jan 13 '16 at 10:08

So I think I've rolled my own with a couple of threads.

For the example below, test.py is this:

#!/usr/bin/python -u

import sys
import time

sys.stdout.write("stdout 1\n")
time.sleep(1)
sys.stderr.write("stderr 2\n")
time.sleep(1)
sys.stdout.write("stdout 3\n")
time.sleep(1)
sys.stderr.write("stderr 4\n")
time.sleep(1)

My code for getting the right output is:

#!/usr/bin/env python

import subprocess
from threading import Thread, Lock

cmdOutput = []
cmdOutputLock = Lock()
STDOUT = 1
STDERR = 2

def _outputLoop( fd, identifier ) :
    line = fd.readline()
    while line :
        cmdOutputLock.acquire()
        cmdOutput.append( ( line, identifier ) )
        cmdOutputLock.release()
        line = fd.readline()

p = subprocess.Popen( "test.py",
                      stdout = subprocess.PIPE,
                      stderr = subprocess.PIPE )

Thread( target=_outputLoop, args=( p.stdout, STDOUT ) ).start()
Thread( target=_outputLoop, args=( p.stderr, STDERR ) ).start()

while fd.poll() is None or cmdOutput :
    output = None
    cmdOutputLock.acquire()
    if cmdOutput :
        output = cmdOutput[0]
        del cmdOutput[0]
    cmdOutputLock.release()

    if output :
        if output[1] == STDOUT :
            print "STDOUT : {}".format( output[0].rstrip() )
        elif output[1] == STDERR :
            print "STDERR : {}".format( output[0].rstrip() )

I can definitely imagine a time when an stderr line is mixed up with an stdout line, but for the sake of what I wanted it for, it certainly works. (I was putting this as part of a logging module which would run a command and use different log levels for stdout and stderr.)

(1) it is still won't preserve order in the general case (2) it is unnecessarily complicated ([compare with this](http://stackoverflow.com/a/31867499/4279)) (3) it may lose data at the end — jfs, Jan 13 '16 at 11:03

score 1 · Answer 3 · edited May 23 '17 at 11:54

1

It is impossible. There is no order defined if writes are performed to different files.

You can get the correct order if the writes to stdout, stderr go to the same place (like in your stdout=PIPE, stderr=STDOUT case).

If "approximate" order is enough; here's a simple code example with threads and here's single-threaded version with a select-loop.

edited May 23 '17 at 11:54

Community

1
1

answered Jan 13 '16 at 10:59

jfs

399,953
195
994
1,670

subprocess.Popen : Separating stdout/stderr but keeping order

3 Answers3