multiprocessing.Queue behavior when forking

Question

After a few os.fork(), I am trying to exchange data to and from the children. To do this, I use multiprocessing.Queue instances. The Queues work correctly when the parent puts and the children get; but not in the opposite way.

My example code:

import os
import multiprocessing as mp
from queue import Empty

if __name__ == '__main__':

    n_workers = 5

    forward_queue = mp.Queue()
    pids_queue = mp.Queue()

    for n in range(n_workers):
        forward_queue.put(n)

    for n in range(n_workers):
        child = os.fork()
        if child:
            pass
        else:
            my_number = forward_queue.get()
            print('pid={} here, my number is {}'.format(os.getpid(), my_number))
            pids_queue.put(os.getpid())
            os._exit(0)  # correct way to exit a fork according to docs

    while True:
        try:
            pid_of_child = pids_queue.get(timeout=5)
        except Empty:
            print('no more pids')
            break
        else:
            print('one of my children had this pid={}'.format(pid_of_child))

the output I get:

pid=19715 here, my number is 0
pid=19716 here, my number is 1
pid=19717 here, my number is 2
pid=19721 here, my number is 3
pid=19718 here, my number is 4
no more pids

the output I would expect:

pid=19715 here, my number is 0
pid=19716 here, my number is 1
pid=19717 here, my number is 2
pid=19721 here, my number is 3
pid=19718 here, my number is 4
one of my children had this pid=19715
one of my children had this pid=19716
one of my children had this pid=19717
one of my children had this pid=19721
one of my children had this pid=19718
no more pids

Can someone explain why this happens?

score 4 · Accepted Answer · answered Jan 20 '20 at 09:03

4

try this, before you exit the fork:

pids_queue.close()
pids_queue.join_thread()

The problem is, how queues work. After you put a value on the queue a background thread is started to transfer the item into the pipe. When you call os._exit immediately, the thread will be closed. For this type of problem, the methods .close and .join_thread are developed.

answered Jan 20 '20 at 09:03

Felix Kleine Bösing

606
3
13

2

I was just writing a similar answer. Note that you'd probably be better off using `multiprocessing.Process()` instead of `os.fork()`. – petre Jan 20 '20 at 09:09
2

@petre Yeah, that would be better. I like to start with a clean process and supply only those objects that are necessary for that process. – Felix Kleine Bösing Jan 20 '20 at 09:18
2

Thank you. I, too, normally go for the cleaner `multiprocessing.Process()`, but the full scale case this example was distilled from really needs to do a naked `os.fork()`. – Lester Jack Jan 20 '20 at 09:21
1

had the same question, did the job, thanks! – B.abba Sep 29 '22 at 13:00

multiprocessing.Queue behavior when forking

1 Answers1