Stack Overflow
Stack Overflow

Questions tagged [concurrent.futures]

concurrent.futures is a Python module which provides a high-level interface for asynchronously executing callables.

The concurrent.futures module aims to provide a simple interface for parallelizing operations in multi-threaded and multi-process Python applications. The module was added to the Python standard library in version 3.2, but a backport is available for Python 2.5+.

929 questions
14
votes
1 answer

gRPC Python thread_pool vs max_concurrent_rpcs

When launching a Python grpc.server, what's the difference between maximum_concurrent_rpcs and the max_workers used in the thread pool. If I want maximum_concurrent_rpcs=1, should I still provide more than one thread to the thread pool? In other…
David Adrian
  • 1,079
  • 2
  • 9
  • 24
14
votes
2 answers

How to monitor python's concurrent.futures.ProcessPoolExecutor?

We are using the ProcessPoolExecutor from concurrent.futures in a service that asynchronously receives requests, and does the actual, synchronous processing in the process pool. Once we ran into the case that the process pool was exhausted, so new…
moritz
  • 12,710
  • 1
  • 41
  • 63
14
votes
1 answer

Individual timeouts for concurrent.futures

I see two ways to specify timeouts in concurrent.futures. as_completed() wait() Both methods handle N running futures. I would like to specify an individual timeout for each future. Use Case: Future for getting data from DB has a timeout of 0.5…
guettli
  • 25,042
  • 81
  • 346
  • 663
14
votes
1 answer

Celery vs. ProcessPoolExecutor / ThreadPoolExecutor

I am creating a django webserver that allows the user to run some "executables" on a local machine and to analyse their output through a webpage. I have previously used a Celery tasks queue in order to run "executables" in similar situations.…
ostrokach
  • 17,993
  • 11
  • 78
  • 90
14
votes
2 answers

Detect failed tasks in concurrent.futures

I've been using concurrent.futures as it has a simple interface and let user easily control the max number of threads/processes. However, it seems like concurrent.futures hides failed tasks and continue the main thread after all tasks…
Erika L
  • 307
  • 4
  • 9
14
votes
1 answer

Is concurrent.futures a medicine of the GIL?

I was just searching about this new implementation, and i use python 2.7, i must install this, so if i use it, i'll forget the word GIL on CPython?
Abdelouahab Pp
  • 4,252
  • 11
  • 42
  • 65
13
votes
1 answer

Copy flask request/app context to another process

tl;dr How can I serialise a Flask app or request context, or a subset of that context (i.e. whatever can be successfully serialised) so that I can access that context from another process, rather than a thread? Long version I have some functions…
daveruinseverything
  • 4,775
  • 28
  • 40
13
votes
1 answer

ProcessPoolExecutor logging fails to log inside function on Windows but not on Unix / Mac

When I run the following script on a Windows computer, I do not see any of the log messages from the log_pid function, however I do when I run on Unix / Mac. I've read before that multiprocessing is different on Windows compared to Mac, but it's not…
blahblahblah
  • 2,299
  • 8
  • 45
  • 60
13
votes
1 answer

Can't pickle coroutine objects when ProcessPoolExecutor is used in class

I'm trying to get asyncio work with subprocesses and limitations. I've accomplish this in functional way, but when I tried to implement same logic in opp style several problems showd up. Mostly Can't pickle coroutine/generator errors. I tracked some…
Alex Smith
  • 311
  • 1
  • 3
  • 14
12
votes
2 answers

AttributeError: module 'concurrent' has no attribute 'futures' when I try parallel processing in python 3.6

I'm trying to split a process that takes a long time to multiple processes using concurrent.futures module. Attached is the code below Main function: with concurrent.futures.ProcessPoolExecutor() as executor: for idx, score in zip([idx for idx in…
Naveen Bharadwaj
  • 325
  • 1
  • 2
  • 12
12
votes
1 answer

How to chain futures in a non-blocking manner? That is, how to use one future as an input in another future without blocking?

Using the below example, how can future2 use the result of future1 once future1 is complete (without blocking future3 from being submitted)? from concurrent.futures import ProcessPoolExecutor import time def wait(seconds): time.sleep(seconds) …
Greg
  • 8,175
  • 16
  • 72
  • 125
12
votes
2 answers

Why is asyncio.Future incompatible with concurrent.futures.Future?

The two classes represent excellent abstractions for concurrent programming, so it's a bit disconcerting that they don't support the same API. Specifically, according to the docs: asyncio.Future is almost compatible with…
max
  • 49,282
  • 56
  • 208
  • 355
12
votes
2 answers

From concurrent.futures to asyncio

I have two problems with concurrent.futures: How to break time.sleep() in a python concurrent.futures? Conclusion: time.sleep() cannot be interrupted. One solution is: You can write a loop around it and do short sleeps. See How to break time.sleep()…
guettli
  • 25,042
  • 81
  • 346
  • 663
12
votes
1 answer

Using `concurrent.futures.Future` as promise

In the Python docs I see: concurrent.futures.Future... ...should not be created directly except for testing. And I want to use it as a promise in my code and I'm very surprised that it is not recommended to use it like this. My use case: I have…
Gill Bates
  • 14,330
  • 23
  • 70
  • 138
12
votes
2 answers

asyncio yield from concurrent.futures.Future of an Executor

I have a long_task function which runs a heavy cpu-bound calculation and I want to make it asynchronous by using the new asyncio framework. The resulting long_task_async function uses a ProcessPoolExecutor to offload work to a different process to…
krdx
  • 1,315
  • 15
  • 22
1 2
3
…
61 62