multiprocessing: how to send data from parent to continuously running child process?

Question

Is it possible to use multiprocessing to send data to a running child process? For example, running a web server in a child process and using the parent process to send data to the server. I found this question about the exact reverse of my situation (i.e. continuous updates from a running child process back to the parent). Is there a way to get this to work? My understanding is that multiprocessing.Queue only sends/receives data on process termination. This pseudocode gives a skeleton of what I'd like to do:

def server(q):
    //start eventlet server
    q.get() // --> does something

q = Queue()
p = Process(target=server, args=(q,))
p.start()
q.put("some kind of command")

Hello rustbird, can you give a little more details as to the context of what you are trying to do exactly. If you have a web server as a child process why not just communicate with it over http. Have you considered using a Task Queue like celery ? https://docs.celeryproject.org/en/stable/getting-started/introduction.html — Omar Nasr, Jul 03 '20 at 00:33

score 2 · Answer 1 · answered Jul 03 '20 at 04:31

2

I think you misread the answer you quoted. The multiprocessing.Queue was designed for exactly the purpose you describe. Maybe check out the reference documentation for that class? An alternative may also be multiprocessing.Pipe.

answered Jul 03 '20 at 04:31

Daniel Junglas

5,830
1
5
22

In my code, the q.get() command is never reached because the eventlet server is still running. I can't figure out how to keep the server running & pass data to it with queues. – rustbird Jul 03 '20 at 18:34

score 2 · Answer 2 · answered Jul 03 '20 at 14:53

Your pseudocode runs almost exactly as written:

from multiprocessing import Queue, Process
import time

def server(q):
    while True:
        stuff = q.get()
        print(stuff, flush=True)

if __name__ == '__main__':
    q = Queue()
    p = Process(target=server, args=(q,))
    p.start()
    for i in range(5):
        q.put(i)
        time.sleep(1)
    p.terminate()

score 0 · Answer 3 · answered May 25 '23 at 04:04

This script creates a parent process that executes a long-running task and a child process that does work in the background. When the parent process is finished, it sends a message to the child to stop, the child receives the message, it stops what it's doing, sends a message back to the parent that it's stopped, the parent receives the message, and then stops itself.

import queue, time
from multiprocessing import Queue, Process

class Parent:
    def __init__(self):
        self.send_signal_queue = Queue()
        self.receive_signal_queue = Queue()

        self.startChild()

        print('Begin long-running task in parent.')
        time.sleep(10)
        print('End long-running task in parent.')

        self.stopChild()

    def startChild(self):
        self.Child = Child(self.send_signal_queue, self.receive_signal_queue)
        self.Child.start()

    def stopChild(self):
        self.send_signal_queue.put('Stop')

        while True:
            try:
                self.receive_signal_queue.get_nowait()
                break
            except queue.Empty:
                time.sleep(1)

        self.Child.terminate()
        self.Child.join()

class Child(Process):
    def __init__(self, receive_signal_queue, send_signal_queue):
        super(Child, self).__init__()
        self.receive_signal_queue = receive_signal_queue
        self.send_signal_queue = send_signal_queue

    def run(self):
        while True:
            try:
                self.receive_signal_queue.get_nowait()
                break
            except queue.Empty:
                pass

            print('Child doing work while parent is busy.')
            time.sleep(1)

        self.send_signal_queue.put('Stopped')

Parent()

multiprocessing: how to send data from parent to continuously running child process?

3 Answers3