python: multiprocessing.Pipe and redirecting stdout

Question

I am using multiprocessing package to spawn a second process from which I would like to redirect stdout and stderr into the first process. I am using multiprocessing.Pipe object:

dup2(output_pipe.fileno(), 1)

Where output_pipe is an instance of multiprocessing.Pipe. However, when I try to read on the other end, it just hangs. I tried reading using Pipe.recv_bytes with a limit, but that raises an OSError. Is this possible at all or should I just switch to some lower level pipe functions?

Can you add a complete, runnable example that demonstrates your error? — JohanL, Feb 09 '18 at 20:59

score 5 · Accepted Answer · answered Feb 09 '18 at 21:59

5

After experimenting in Python 2.7 I got this working example. With os.dup2 pipe's file descriptor is copied to standard output file descriptor, and each print function ends up writing to a pipe.

import os
import multiprocessing


def tester_method(w):
    os.dup2(w.fileno(), 1)

    for i in range(3):
        print 'This is a message!'


if __name__ == '__main__':
    r, w = multiprocessing.Pipe()

    reader = os.fdopen(r.fileno(), 'r')

    process = multiprocessing.Process(None, tester_method, 'TESTER', (w,))
    process.start()

    for i in range(3):
        print 'From pipe: %s' % reader.readline()

    reader.close()
    process.join()

Output:

From pipe: This is a message!

From pipe: This is a message!

From pipe: This is a message!

answered Feb 09 '18 at 21:59

Ilija

1,556
1
9
12

Hi, It indeed works, however when reading from pipe as: `r.recv()` it does not. Do you know why, or how to check (non blocking) if reader has anything to read? – Radoslaw Garbacz Nov 19 '18 at 23:33
@RadoslawGarbacz You must use `w.send` then, as per https://docs.python.org/2/library/multiprocessing.html#Connection.recv. – Ilija Nov 21 '18 at 10:02
getting this error - OSError: [Errno 9] Bad file descriptor for os.fdopen statement. Any idea how to fix this? – James Gopinath Feb 07 '19 at 09:49
1

Note - Just now tried in Linux, this code works, whereas the same code is failing in Windows - OSError: [Errno 9] Bad file descriptor – James Gopinath Feb 07 '19 at 11:04
I tried it on Windows, it gave OSError: [Errno 6] The handle is invalid. – pansila Sep 24 '19 at 07:34

score 0 · Answer 2 · answered Dec 18 '21 at 04:10

The existing answer works for the raw file descriptors, but this may be useful for using Pipe.send() and recv:

    class PipeTee(object):
        def __init__(self, pipe):
            self.pipe = pipe
            self.stdout = sys.stdout
            sys.stdout = self

        def write(self, data):
            self.stdout.write(data)
            self.pipe.send(data)

        def flush(self):
            self.stdout.flush()

        def __del__(self):
            sys.stdout = self.stdout

To use this, create the object in your multiprocess function, pass it the write side of multiprocessing.Pipe, and then use the read side on the parent process with recv, using poll to check if data exists.

python: multiprocessing.Pipe and redirecting stdout

2 Answers2

Linked