Multithreading in Python: Processing Similar to Producer Consumer

Question

In Python 2.7, how can one know when a thread has finished such that one can maintain a constant buffer of running threads, at least until all processing has finished?

I am trying to keep a buffer of threads (similar to a producer/consumer problem but treating the threads themselves as the resources in the buffer, and a finished thread as being the resource ready for consumption) that, when a thread finishes, puts a new thread in the buffer to start and I can't seem to figure out how this can be done using the threading module and Queues.

score 1 · Answer 1 · answered Nov 05 '14 at 17:03

It's more common to start a small number of threads which live forever than to start a lot of threads one after the other. Try writing your thread code like this:

def run(q):
    # Threads run this function
    # q is a Queue
    while True:
        work = q.get()
        process(work)
        q.task_done()

Multithreading in Python: Processing Similar to Producer Consumer

1 Answers1