Python Thread Wait() Timeout For Queue .join()

Question

In python, I have multiple threads running and I need the main process to wait until they are done, so I did this with Queue class .join() method. However, I wanted to implement SIGINT but the handler for it wouldn't execute because join() was blocking it(the threading processes run for at least 5 minutes for what I have them doing). So I modified Queue's .join() and placed a time out in wait():

class CustomQueue(Queue.Queue):
    #Can not use .join() because it would block any processing
    #for SIGINT untill threads are done. To counter this,
    # wait() is given a time out along with while not kill_received
    #to be checked

    def join(self):
        self.all_tasks_done.acquire()
        try:
            while not kill_received and self.unfinished_tasks:
                self.all_tasks_done.wait(10.0)
        finally:
            self.all_tasks_done.release()

This works beautifully and perfect for me.

But what I don't understand is the time out in wait(). For instance, I should not be able to send a SIGINT and have it process for at least 10 seconds. But, I am able to in less than 10 seconds. It doesn't matter what the seconds are, the SIGINT handler function is able to process without being blocked. Why is this? I should have to wait at least 10 seconds for the wait to time out and self.all_tasks_done.release() to run so the SIGINT function will process...rather than the SIGINT function processing before the wait() time out.

score 2 · Answer 1 · answered Dec 01 '13 at 01:34

2

We're missing information here that may be important:

Which version of Python?
Which OS?

It may be important because mixing threads with signals is a cross-platform mess. CPython tries to make some sense of it all, but has had various degrees of success across various Python versions and OSes.

Anyway, the answer to your question may be simple ;-) Queue.all_tasks_done is a threading.Condition, and Condition implements .wait(timeout) (eventually, drilling down) using time.sleep(). As documented,

time.sleep(secs) Suspend execution for the given number of seconds. ... The actual suspension time may be less than that requested because any caught signal will terminate the sleep() following execution of that signal’s catching routine. ...

By default SIGINT raises KeyboardInterrupt. So if you don't have a handler installed for SIGINT, SIGINT terminates the sleep() early and raises KeyboardInterrupt. If you do have a SIGINT handler installed, SIGINT will still terminate the sleep() early, but what happens after that depends on what your handler does.

answered Dec 01 '13 at 01:34

Tim Peters

67,464
13
126
132

For "time.sleep(secs) Suspend execution for the given number of seconds", is this suspending all tasks in main and only allowing the multi-thread tasks to run? – dman Dec 01 '13 at 02:12
1

`time.sleep()` only affects the thread from which it is invoked. So, sure, if you invoked it from the main thread, the main thread will block there (until a timeout or signal wakes it), but other threads are unaffected. Nothing special bout the main thread here, *except that* Python delivers all signals to the main thread. – Tim Peters Dec 01 '13 at 02:29
I see now what wait() does. I am confused by the condition variable "all_tasks_done". Doesn't the condition object only apply to one thread at a time? Why do the call the condition object "all_tasks_done"? It only has aqcuire, release, wait, and notify, notify_all. – dman Dec 01 '13 at 04:44
1

Sorry, I don't understand your question (no idea what "only apply to one thread at a time" could mean), and an SO comment is far too constrained to write a tutorial on Conditions ;-) [Here's a good intro](http://pages.cs.wisc.edu/~remzi/OSTEP/threads-cv.pdf), for the related `pthreads` condition variables. Proper use of Conditions requires a real learning curve, but they make it very much easier to write correct parallel-safe code. – Tim Peters Dec 01 '13 at 04:52
Also, what happens if all_tasks_done.wait(10.0) times out and there is a another thread that has the lock? Will the thread go back into wait() again after it tries acquire(this is what it does when it goes out of wait(), right)? – dman Dec 01 '13 at 05:31
1

It is impossible for `.wait()` to return without acquiring the Condition's lock. That's part of what makes Conditions so powerful. In correct use of Condition, nobody *ever* holds the lock for "a long time", so there's nothing to worry about here for correct uses. Please open new issues for new questions, OK? This has nothing to do with your original question, and after a few more comments SO will throw this into a chat room - long chains of comments aren't how SO is intended to work. – Tim Peters Dec 01 '13 at 05:35

Python Thread Wait() Timeout For Queue .join()

1 Answers1