Flushing a pipe (os.pipe) before closing

Question

I need to launch a subprocess and enable two threads for reading its stdout and stderr respectively.

The following code is just considering stdout:

def reader(rfd):
    while True:
        try:
            data = os.read(rfd, bufsize)
        except OSError:
            break
        else:
            chomp(data)

rout, wout = os.pipe()
tout = threading.Thread(target=reader, args=(rout,))
tout.start()

subprocess.check_call(command, bufsize=bufsize, stdout=wout, stderr=werr)

os.close(wout)
os.close(rout)
tout.join()

The code works, except I noticed that not all data is processed, as if the os.close(wout) function kills the reader before all data is read. On the other hand, if I don't close wout my process will be hanging forever on tout.join().

I can tell this is a buffering problem because if I put a very bad time.sleep(0.1) just after subprocess.check_call(...) everything magically works.

The good way would be flushing instead of waiting, but any call to os.fsync() over a pipe is giving OSError: [Errno 22] Invalid argument.

Any hint about how to flush a pipe created with os.pipe?

@goncalopp, the `close` interrupts the `read`. If you do the opposite, you get a deadlock, since the thread never terminates. — Dacav, Sep 16 '14 at 10:29
Have you tried using `subprocess.Popen` with `stdout=subprocess.PIPE`? Also, are you sure it is not the call to close `rout` that is the problem? — Dunes, Sep 16 '14 at 10:40
@Dunes, I refrained from doing that since I know communicate buffers the whole content (in my case that would be huge). But now I realize that PIPE can be specified even when not using it. I'll try. If this works I'll let you know. About closing `rout`, if I don't the thread gets stuck on read. — Dacav, Sep 16 '14 at 10:45
@Dacav Yes, of course - but it will process all the data, right? You just need to wait for the process to finish, and close the write ends. I'll put that in a answer in a bit — loopbackbee, Sep 16 '14 at 10:52
@Dunes, Thanks, it worked. If you put that as an answer I will accept it. — Dacav, Sep 16 '14 at 11:00
@goncalopp, I solved it, but if you put an answer I will upvote it. — Dacav, Sep 16 '14 at 11:00

score 2 · Accepted Answer · answered Sep 16 '14 at 11:04

I would recommend using Popen rather than os.pipe for interprocess communication.

eg.

writer_process = subprocess.Popen(cmd, stdout=subprocess.PIPE)
reader_thread = threading.Thread(target=reader, args=(writer_process.stdout,))
reader_thread.start()
reader_thread.join()

However, if you really want to use os.pipe then you will have an easier time treating them like file objects. Python's in-built file context manager will make sure the files and flushed and closed appropriately.

eg.

def reader(fd):
    with os.fdopen(fd, bufsize=bufsize) as f:
        while True:
            data = f.read(bufsize)
            if not data:
                break
            chomp(data)

and

with os.fdopen(wout, "w", bufsize=bufsize) as f:
    subprocess.check_call(cmd, stdout=f)

score 1 · Answer 2 · answered Sep 16 '14 at 11:00

You can't close the pipes until you're sure the process is done writing (because you'll lose data), and you can't wait for the thread to finish without closing the pipes (because os.read will block forever).

You need to wait for process completion, and close the writing end of the pipe manually (since you created it).

Here's a self contained example:

import os
import threading
import subprocess

def reader(rfd):
    while True:
        try:
            data = os.read(rfd, bufsize)
            if not data:
                break #reached EOF
        except OSError:
            break

bufsize=100

rout, wout = os.pipe()
rerr, werr = os.pipe()

tout = threading.Thread(target=reader, args=(rout,))
tout.start()

p= subprocess.Popen("ls", bufsize=bufsize, stdout=wout, stderr=werr)
p.wait() #wait for the proces to end writing
os.close(wout)
os.close(werr)
tout.join()
os.close(rout)
os.close(rerr)

Flushing a pipe (os.pipe) before closing

2 Answers2