Python - Using parallel processes communicating with API to improve speed

Question

I'm trying to write a program in which 1 function adds information to a queue and the other function reads the queue in the meanwhile and does some misc task with it. The program has to put info in the queue and read from it at the same time.

Example:

from multiprocessing import Process
from time import clock, sleep
import Queue

class HotMess(object):
    def __init__(self):
        self.market_id=['1','2','3']
        self.q2=Queue.Queue()
        self.session=True

    def run(self):
        if __name__=='__main__':
            t = Process(target=self.get_data)
            k = Process(target=self.main_code)
            t.run()
            k.run()

    def main_code(self):
       while self.session:
        result=self.q2.get()
        print(result)

    def get_data(self):
        #while self.session:
        for market in self.market_id:
            self.q2.put(market)

mess=HotMess()
mess.run()

Now this produces output of 1 2 3. So far so good. But I actually want get_data to be a while loop and basicly run indefinately. Now if you uncomment while self.session: in the get_data function and do the indent. Now it doesn't produce any output and I think this is due that the Process get_data doesn't finish as long as self.session=True.

My question is how can I main_code() not wait for get_data() and just start working the Queue so they both interact with the Queue (q2). I tried looking at process/threading/POpen but i'm quite far out of my comfort zone and at a bit of a loss.

score 0 · Answer 1 · edited May 23 '17 at 11:51

look here

you should use multiprocessing.Queue. it's for the communication between different processes.

And I also change run to start and join.

from multiprocessing import Process, Queue
from time import clock, sleep

class HotMess(object):
    def __init__(self):
        self.market_id=['1','2','3']
        self.q2=Queue()
        self.session=True

    def run(self):
        if __name__=='__main__':
            t = Process(target=self.get_data)
            k = Process(target=self.main_code)
            t.start()
            k.start()
            t.join()
            k.join()

    def main_code(self):
        while self.session:
            result=self.q2.get()
            print(result)

    def get_data(self):
        while self.session:
            for market in self.market_id:
                self.q2.put(market)

mess=HotMess()
mess.run()

Python - Using parallel processes communicating with API to improve speed

1 Answers1