Python using queues for countdown watchdog timer

Question

I have a program which spawns 4 threads, these threads need to stay running indefinitely and if one of them crashes I need to know so I can restart.

If I use a list with 4 numbers and pass it to each thread through using a queue. Then all each thread has to do is reset its section in the timer while the main thread counts it down.

So the queue will never be empty, only a single value could go to 0, and then if this happens then the main thread knows its child hasn't responded and it can act accordingly.

But every time I .get() from the queue, it makes it empty, so I have to get from the queue, store into a variable, modify the variable and put it back in the queue.

Is this fine using the queue like this for a watchdog.

quodlibetor · Answer 1 · 2012-07-01T21:45:47.720

If you're using Threads, you could regularly check through threading.enumerate to make sure that you have the correct number and kind of threads running.

But, also, passing things into a Queue that gets returned from a thread is a technique that I have at least seen used to make sure that threads are still running. So, if I'm understanding you correctly, what you're doing isn't completely crazy.

Your "thread must re-set its sentinal occasionally" might make more sense to have as a list of Queues that each Thread is expected to respond to asap. This depends on if your Threads are actually doing process-intensive stuff, or if they're just backgrounded for interface reasons. If they're not spending all their time doing math, you could do something like:

def guarded_thread(sentinal_queue, *args):
    while True:
        try:
            sentinal_queue.get_nowait()
            sentinal_queue.put('got it')
        except Queue.Empty:

            # we just want to make sure that we respond if we have been
            # pinged
            pass

        # do actual work with other args

def main(arguments):
    queues = [Queue() for q in range(4)]
    threads = [(Thread(target=guarded_thread, args=(queue, args)), queue)
               for queue, args in zip(queues, arguments)]

    for thread, queue in threads:
        thread.start()

    while True:
        for thread, queue in threads:
            queue.put(True)

        for thread, queue in threads:
            try:
                response = queue.get(True, MAX_TIMEOUT)
                if response != 'got it':
                    # either re-send or restart the thread
            except Queue.Empty:
                # restart the thread
        time.sleep(PING_INTERVAL)

Note that you could also use different request/response queues to avoid having different kinds of sentinal values, it depends on your actual code which one would look less crazy.

But what if a thread hangs? I can keep checking to see if their alive, or if the enumerate count is correct, but if a thread hangs then it will just stay open and I won't know that it has stopped working. — user185955, Jul 01 '12 at 20:02
Oh, yes, in that case having a guard value or queue that you ping them with occasionally makes sense. — quodlibetor, Jul 01 '12 at 20:05
what do you mean guard value? what would be the best way to go about what I need to do? — user185955, Jul 01 '12 at 20:09
Check my code, it's broken and untested, but it's the general technique I'd use for doing what you're interested in. — quodlibetor, Jul 01 '12 at 20:29

Python using queues for countdown watchdog timer

1 Answers1