What is the significance of condition.notify() in python's threading module?

Question

So I'm using python's Condition from the threading module:

from threading import Thread, Condition
condition = Condition()

I have a Producer class (subclass of Thread) that essentially in a for loop adds items to a queue until the queue is full (i.e. reached a defined max length) and a Consumer class that in a for loop pops items unless the queue is empty. In the Producer, if the queue is full, we have a condition.wait() statement, and similarly in the consumer class, if the queue is empty we have a condition.wait() statement. If neither of these conditions are met (queue neither full nor empty) each class do its thing (adds an item to the queue or pops an item, for Producer or Consumer respectively) and then before releasing the condition ( condition.release()) , we have a condition.notify() statement. I read from the documentation that notify() wakes up one of the threads waiting.

My question now is two folds:

1. What does exactly "waking up" mean for a thread?
2. When I removed the notify() statements from both classes, my program runs just fine for few iteration of the for loops (i.e. producer pushes items and consumer pops items), and then at some point the producer keeps pushing items to the queue without consumer running, the queue gets full, but the consumer never runs again and the program just sorta halts. What is the significance of notify() that seems essential to the performance of this program.

Thanks a lot for your help:)

Answer 1

1) It signals thread waiting for a lock release it can continue execution.

2) Condition.notify(n) takes up to n threads from the internal queue and calls release on the locks they are waiting for - thus waking them up. If the internal queue is empty, there is no one to wake up and the notify call has no effect. That's why from the beginning, removing notify had no effect, but once consumer threads called wait, there was no one to wake them up and they were waiting for ever.

Answer 2

TL;DR

1. Waking up here can be seen as rousing the producer or consumer from sleep AND informing them that they can get back to work as soon as their shared PPE (the underlying lock) is available.

2. Consumer saw when the queue got empty, and went to sleep, waiting for the producer to wake it up when it adds to the queue.

   But you've taken away the producer's ability to wake the consumer up.

   Producer saw when the queue got full, and went to sleep, waiting for the consumer to wake it up when it takes from the queue.

   But you've taken away the consumer's ability to wake the producer up.

   And now they're both asleep.

---

The Discussion Proper

The key to appreciating the significance of Condition.notify() lies in realizing that when you're wait()ing on a condition, you are NOT waiting for its underlying lock to become available, but rather, you're waiting for some notification or a timeout on that condition.

According to the Python docs on threading.Condition.wait

wait(timeout=None)
    Wait until notified or until a timeout occurs...

When you invoke notify on a condition, you're essentially informing n threads which invoked wait on that condition object that they may stop waiting as soon as they can acquire the underlying lock.

This can be inferred from the Python docs on threading.Condition.notify

notify(n=1)
    ...
    Note: an awakened thread does not actually return
    from its wait() call until it can reacquire the lock.
    Since notify() does not release the lock, its caller should.

Now, here's my educated guess at what is happening with your code:

• Producer pushes a number of items onto queue.
• Consumer consumes a number of items from queue.
• Queue soon becomes empty.
• Consumer wait()s to be notified that it may resume consuming (as soon as it can acquire the underlying lock)
• Producer pushes items onto queue.
• Producer never gets to notify() consumer that it may resume consuming.
• Queue soon becomes full.
• Producer wait()s to be notified that it may resume producing (as soon as it can acquire the underlying lock)
• Consumer is waiting to be awakened by Producer, and Producer is waiting to be awakened by Consumer.
• Stalemate!