How to wait and retrieve the result in producer-consumer pattern?

Question

I have an application which uses producer-consumer pattern: multiple producers, one consumer. The catch here is that inside each producer, after submitting the task, I want to wait and retrieve the result inside the same producer process:

from multiprocessing import Process
from queue import Queue


def producer(queue, work_item):
    queue.put(work_item)

    # ??? How to wait for the work_item to be done, and get back the result of processing work_item???

    # "result" is the result of producer processing this item. I want to print the result "INSIDE THE PRODUCER PROCESS"
    print("the result of processing {} is {}".format(work_item, result))


def consumer(queue)    :
    while True:
        item = queue.get()
        # process the item
        result = process(item)
        # ??? How to pass this result back to producer???
        # processing done
        queue.task_done()


if __name__== "__main__":
    queue=Queue()
    p1 = Process(target=producer, args=(queue,1))
    p2 = Process(target=producer, args=(queue,2))
    p3 = Process(target=producer, args=(queue,3))

    c1 = Process(target=consumer, args=(queue,))

    p1.start()
    p2.start()
    p3.start()
    c1.start()

Why I want to put task submission and result retrieval inside the same producer?

Because in my actual application:

the producer could be a web request handler in Flask web server,
the consumer could be an tensorflow image classifier.

In each web request, user submit an image for classification, but classification is an time-consuming task, so the web request handler (my producer) then delegate this task to the tensorflow image classifying process (my consumer) to do the work, but the web request handler must wait for the result, and pass it back to user.

So how to wait and retrieve result in producer-consumer pattern?

ps. I have already tried Celery. But here I am more interested in a Core Python implementation (or general concurrency pattern), if there is any.

Make `producer`s actually put promises in the queue and `await` for them to be fulfilled? — bipll, Dec 05 '18 at 13:43
@bipll `promises` as in asyncio promises? That might work. Let me look for some minimal viable example or try to write one to verify. — modeller, Dec 05 '18 at 13:51
Let me suggest an alternative approach. Now you basically expect client to wait (possibly indefinitely) for a response. How about return some job identifier to client right away and then either push notification to the client or let the client ask about the result in next call? — freakish, Dec 05 '18 at 13:57
@freakish Or let client pass one more argument to `consumer` which is a callback called once the item is processed. — bipll, Dec 05 '18 at 15:52

score 1 · Answer 1 · answered Dec 07 '18 at 12:58

I would use an event that halts your producers until your consumer is available again. Addtionally, I would limit the size of the queue to avoid mixing of results between the tasks. In your example it would look like the following (not tested). However, there might be the risk of a race condition...

from multiprocessing import Process, Event
from queue import Queue


def producer(queue, work_item, yoursignal):
    # do stuff
    if yoursignal.is_set():
        yoursignal.clear() # block others
        queue.put(block=True)
        yoursignal.wait() # wait for signal to become True
        print("the result of processing {} is {}".format(work_item, result))


def consumer(queue, yoursignal):
    while True:
        item = queue.get(block=True) # wait until something is available
        # process the item
        result = process(item)
        # ??? How to pass this result back to producer???
        # processing done
        queue.task_done()
        yoursignal.set()


if __name__== "__main__":
    queue = Queue(1) # only one slot in the queue
    signal = Event()
    p1 = Process(target=producer, args=(queue,1, signal))
    p2 = Process(target=producer, args=(queue,2, signal))
    p3 = Process(target=producer, args=(queue,3, signal))

    c1 = Process(target=consumer, args=(queue, signal))

    p1.start()
    p2.start()
    p3.start()
    c1.start()

How to wait and retrieve the result in producer-consumer pattern?

1 Answers1