Checking up on a `concurrent.futures.ThreadPoolExecutor`

Question

I've got a live concurrent.futures.ThreadPoolExecutor. I want to check its status. I want to know how many threads there are, how many are handling tasks and which tasks, how many are free, and which tasks are in the queue. How can I find out these things?

I don't believe those operations are part of the API. You will likely have to did into the source to poke at those internals: http://hg.python.org/cpython/file/default/Lib/concurrent/futures/thread.py http://hg.python.org/cpython/file/default/Lib/concurrent/futures/process.py — hughdbrown, Aug 24 '14 at 17:47
Right now there's no way to tell which threads are busy. There's a TODO in the code that notes that ability should be added so that new threads aren't created to handle tasks if there are already idle threads which could handle them. — dano, Aug 24 '14 at 18:40

johntellsall · Accepted Answer · 2014-08-25T01:27:26.123

14

There is some visibility into the Pool, and the pending workitem queue. To find out what's available, print poolx.__dict__ to see the structure. Read the ThreadPool code, it's pretty good: concurrent.futures.thread

The following creates a pool with one thread. It then creates two jobs: one sleeps for 3 seconds, the other immediately returns. The pool's number of pending work items is then printed.

Following that, we print out items from the work queue. In this case, a thread is already executing the time.sleep(3) function, so that's not in the queue. The function sleep with args [0] and kwargs {} is printed, because that's the next work item for the pool to run.

Kudos to @dano for the nondestructive queue insight, and @abarnert.

source

import concurrent.futures, time

poolx = concurrent.futures.ThreadPoolExecutor(max_workers=1)
poolx.submit(time.sleep, 3)
poolx.submit(time.sleep, 0)   # very fast

print('pending:', poolx._work_queue.qsize(), 'jobs')
print('threads:', len(poolx._threads))
print()

# TODO: make thread safe; work on copy of queue?
print('Estimated Pending Work Queue:')
for num,item in enumerate(poolx._work_queue.queue):
    print('{}\t{}\t{}\t{}'.format(
        num+1, item.fn, item.args, item.kwargs,
        ))

poolx.shutdown(wait=False)

output

pending: 1 jobs
threads: 1

Pending Work Queue:
1   <built-in function sleep>   (0,)    {}

edited Aug 25 '14 at 01:27

answered Aug 24 '14 at 18:23

johntellsall

14,394
4
46
40

1

You can check what's in the queue without destroying it : `for item in poolx._work_queue.queue: print(item.fn, item.args, item.kwargs)`. – dano Aug 24 '14 at 18:32
Woo thanks again @dano -- I've updated the code and description to add your insight. – johntellsall Aug 24 '14 at 18:43
3

This isn't thread safe. The `pool._work_queue` part is fine, but iterating over `pool._work_queue.queue` is not. That queue member is only supposed to be used with appropriate synchronization; your iterator could be invalidated at any point by another thread pushing or popping. You'll probably _usually_ get away with it in CPython in low-contention scenarios, but with a GIL-less implementation or high contention (i.e., exactly the cases you really need a queue) it'll fail. – abarnert Aug 24 '14 at 22:02
It seems like getting the queue size should be at least safe, which is enough to answer about 75% of the question... But to know what's queued, you really need to explicitly hand it a safely-iterable queue, not use the one in queue.Queue. – abarnert Aug 24 '14 at 22:09
1

@abarnert I suppose you could just acquire `pool._work_queue.mutex` prior to iterating over it. That's all `queue.Queue` uses for synchronization. – dano Aug 25 '14 at 17:31
@dano: If you only care about CPython, and only a specific range of versions you've checked, then yeah. But for other implementations, you may need to do something different—or it may just not be possible. (If Jython or Iron wraps a JVM or .NET lock-free queue, for example—which would be a good idea, but I don't know if they actually do—this either won't be impossible at all, or will have to be written very differently…) – abarnert Aug 25 '14 at 19:17
@shavenwarthog: I don't know that "work on copy of queue" would help here. Is there a synchronized copy operation? If not, it'll just shrink your race condition, making it harder to debug, without eliminating it. – abarnert Aug 25 '14 at 19:18
1

anybody know what version of python this covers? I'm trying it now and getting an attribute error on ._work_queue.queue. AttributeError: '_queue.SimpleQueue' object has no attribute 'queue' – smokes2345 Apr 18 '22 at 15:18

score 0 · Answer 2 · answered Jul 24 '19 at 18:00

Nor very clean and reliable way to find pending futures, but I do it like this:

if 'state=pending' in str(future):
    logger.debug('PENDING')
elif future.running():
    logger.debug('RUNNING')
elif future.cancelled():
    logger.debug('CANCELLED')
elif future.exception():
    logger.debug('EXCEPTION')
elif future.done():
    logger.debug('DONE')

Checking up on a `concurrent.futures.ThreadPoolExecutor`

2 Answers2

source

output

Linked