In Python, how do you get data back from a particular process using multiprocessing?

Question

import multiprocessing as mp
import time

def build(q):
    print 'I build things'
    time.sleep(10)
    #return 42
    q.put(42)

def run(q):
    num = q.get()
    print num

    if num == 42:
        print 'I run after build is done'
        return
    else:
        raise Exception("I don't know build..I guess")

def get_number(q):
    q.put(3)

if __name__ == '__main__':
    queue = mp.Queue()

    run_p = mp.Process(name='run process', target=run, args=(queue,))
    build_p = mp.Process(name='build process', target=build, args=(queue,))
    s3 = mp.Process(name='s3', target=get_number, args=(queue,))

    build_p.start()
    run_p.start()
    s3.start()

    print 'waiting on build'
    build_p.join(1) # timeout set to 1 second
    s3.join()

    print 'waiting on run'
    run_p.join()

    queue.close()
    print 'waiting on queue'
    queue.join_thread()
    print 'done'

My goal is to send build and run into different workers, and run has to get result back from build in order to proceed.

The above revised code based on your help will actually return exception, because s3 is returned before build has the chance.

The value in the front of the queue is now 3. How can we make sure we get the answer back from build process?

Thanks.

I think there is probably a better way to accomplish what you need. Can you explain the problem you are trying to solve ? — Nix, Jun 26 '12 at 22:40
Still doesn't make sense to me and it looks like bad use of multiprocessing bc the logic is synchronus... some of those processes are not needed, i am assuming you have more logic to run then just returning a number? — Nix, Jun 26 '12 at 22:49
@Nix Yes. I am just showing that I have three functions running in the background. Two of which are `Build` and `Run`. `Run` depends on the return values from `Build`, which is 42 in this case. But all three functions are sharing one queue. If process 3 finishes before Build does, `Run` will just take the return value from proc3 and use it, which happens to be `3` rather than the correct value `42`. — user1012451, Jun 26 '12 at 22:51
Yes and NO. You can think of this as a compiler program. You need to compile or build your source code, and return a status code before you knew "oh I can now run my .exe file`. But the user can run an older build which has finished already. In that case, you will have multiple processes running. If that's that you mean by parallel. Sorry for being very obscure. — user1012451, Jun 26 '12 at 22:56
So if you read a value from the queue and its not 42 what do you do with it? Discard? In my program i created a loop i run, and it will run until it gets the right number... — Nix, Jun 26 '12 at 22:58
@Nix You are right. Maybe this is not the best way to handle what I want to do. I will make a new question based on my new requirement. But I've learned a lot from your help. — user1012451, Jun 26 '12 at 23:57

Nix · Accepted Answer · 2012-06-26T22:50:46.707

Your question is a little murky..the problem you are describing sounds synchronous so 3 processes are a little overkill.

Assuming you are just trying to pass values to run you could use the queue object.

import multiprocessing as mp
import time

def build(q):
    print 'I build things'
    time.sleep(5)
    q.put(42)
    return 

def run(q):
    while True:
        num = q.get()
        if num == 42:
            print 'I run after build is done'
            return
        else:
            print 'not the right number...'

    def get_number():
        return 41

if __name__ == '__main__':
    queue = mp.Queue()

    run_p = mp.Process(name='run process', target=run, args=(queue,))
    build_p = mp.Process(name='build process', target=build, args=(queue,))

    run_p.start()
    build_p.start()

    print 'waiting on build'
    build_p.join()
    print 'waiting on run'
    run_p.join()
    queue.close()
    print 'waiting on queue'
    queue.join_thread()
    print 'done'

Thank you very much. It is very helpful. But just again, how do we know the answer we want in the queue right now is `42`? See the above edit :) Thanks! — user1012451, Jun 26 '12 at 22:46

In Python, how do you get data back from a particular process using multiprocessing?

1 Answers1

Linked