queue.Queue() or multiprocessing.Queue() for threads within a Process

Question

Say I have 3 Processes:

In Process A, there is an infinite loop that does something with self.marker. But at the same time, self.marker also needs to be updated every 1 minute so a threading.Timer is implemented to do this.

This Timer will be running within Process A as a separate thread like so:

class A(Process):
    def _init__(self):
        self.marker = True
        self.q = Queue()

    def run(self):
        threading.Timer(60, updater).start()

        while True:
            if not self.q.empty():
                item = self.q.get()
                self.marker = item

            print(self.marker)

    def updater(self):
        self.q.put(not self.marker)
        threading.Timer(60, updater).start()

class B(Process):
    def _init__(self):

    def run(self):

class C(Process):
    def _init__(self):

    def run(self):

a = A()
b = B()
c = C()
a.start()
b.start()
c.start()

My questions are:

1) In this code, is self.marker being safely updated through the use of a queue? Or is a queue unnecessary

2) Should self.q be a queue.Queue() or a multiprocessing.Queue?

You are not doing any multiprocessing, thus you do not need that type of queue. There is another race condition that you do not protect yourself againt (even if it is very unlikely). Since you are updating `self.marker` in `run()` and adding it to the queue in `updater()` you _could_ end up with the same value being written to the que multiple times in a row. (Say the OS decides to swap out `run()` just before `self.marker = item` and does some very heavy computation elsewhere and then come back to swap in `updater()` (which is done sleeping). Then `updater()` will update with the old value. — JohanL, Mar 19 '18 at 06:30
@JohanL Good point. It's actually even worse than your example. Suppose you insert some code between `item = self.q.get()` and `self.marker = item`. And suppose the code takes longer than 60 seconds. In that case updater() is *guaranteed* to post the same value twice. Fiddling with the same variable in two different threads like the OP is doing is asking for trouble. — Paul Cornelius, Mar 19 '18 at 09:51

score 0 · Answer 1 · answered Mar 19 '18 at 05:18

The answer to your first question is: not quite. Timers create secondary threads, so Process A actually has two threads. The function updater runs in the secondary (timer) thread while the run function executes in the main thread. updater accesses the value of self.marker and run sets the same variable, which is a possible race condition. It will be OK if the only thing you're doing with the queue is handling those timer events. But if there is other code writing into the queue you might have a problem: self.marker might change in the moments between the put statement in updater and the get statement in run.

The answer to the second question isn't clear from what you've posted. A thread.Queue() simply moves the same python object from one end of the queue to the other. A multiprocessing.Queue() pickles the object on the input end and does an unpickle on the output, thus recreating the object and allowing it to be shared across Processes. (Objects aren't shared between Processes, but they are between threads.)

BTW you should probably put a time.sleep() into run or else your print statement will bombard you with tons of output.

queue.Queue() or multiprocessing.Queue() for threads within a Process

1 Answers1