How to redirect the stdout of a multiprocessing.Process

Question

I'm using Python 3.7.4 and I have created two functions, the first one executes a callable using multiprocessing.Process and the second one just prints "Hello World". Everything seems to work fine until I try redirecting the stdout, doing so prevents me from getting any printed values during the process execution. I have simplified the example to the maximum and this is the current code I have of the problem.

These are my functions:

import io
import multiprocessing
from contextlib import redirect_stdout


def call_function(func: callable):
    queue = multiprocessing.Queue()                                      
    process = multiprocessing.Process(target=lambda:queue.put(func()))
    process.start()          

    while True:
        if not queue.empty():
            return queue.get()


def print_hello_world():
    print("Hello World")

This works:

call_function(print_hello_world)

The previous code works and successfully prints "Hello World"

This does not work:

with redirect_stdout(io.StringIO()) as out:
    call_function(print_hello_world)
print(out.getvalue())

With the previous code I do not get anything printed in the console.

Any suggestion would be very much appreciated. I have been able to narrow the problem to this point and I think is related to the process ending after the io.StringIO() is already closed but I have no idea how to test my hypothesis and even less how to implement a solution.

Camilo Andrés Ospina · Answer 1 · 2019-08-04T04:49:08.520

This is the workaround I found. It seems that if I use a file instead of a StringIO object I can get the things to work.

with open("./tmp_stdout.txt", "w") as tmp_stdout_file:
    with redirect_stdout(tmp_stdout_file):
        call_function(print_hello_world)
    stdout_str = ""
    for line in tmp_stdout_file.readlines():
        stdout_str += line
    stdout_str = stdout_str.strip()

print(stdout_str)  # This variable will have the captured stdout of the process

Another thing that might be important to know is that the multiprocessing library buffers the stdout, meaning that the prints only get displayed after the function has executed/failed, to solve this you can force the stdout to flush when needed within the function that is being called, in this case, would be inside print_hello_world (I actually had to do this for a daemon process that needed to be terminated if it ran for more than a specified time)

sys.stdout.flush()  # This will force the stdout to be printed

How to redirect the stdout of a multiprocessing.Process

1 Answers1