Stack Overflow
Stack Overflow

Questions tagged [concurrent.futures]

concurrent.futures is a Python module which provides a high-level interface for asynchronously executing callables.

The concurrent.futures module aims to provide a simple interface for parallelizing operations in multi-threaded and multi-process Python applications. The module was added to the Python standard library in version 3.2, but a backport is available for Python 2.5+.

929 questions
8
votes
1 answer

Using concurrent.futures to consume many dequeued messages a time

I'm consuming messages from a RabbitMQ channel, I wish I could consume n elements at a time. I think I could use a ProcessPoolExecutor (or ThreadPoolExecutor). I just wonder if it's possible to know if there's a free executor in the pool. This is…
Zen
  • 917
  • 1
  • 7
  • 20
8
votes
1 answer

ThreadPoolExecutor + Requests == deadlock?

I have a tiny stupid code, which makes a lot of requests to google search service from concurrent.futures import ThreadPoolExecutor import requests import requests.packages.urllib3 requests.packages.urllib3.disable_warnings() def check(page): r…
Anton Hulikau
  • 336
  • 1
  • 3
  • 7
8
votes
1 answer

Valid futures vs Default constructed futures

I am studying futures in my concurrent programming class. My professor has stated this in her slides: "Valid" futures are future objects associated to a  shared state, and are constructed by calling one of the following…
Flame of udun
  • 2,136
  • 7
  • 35
  • 79
7
votes
2 answers

Python ProcessPoolExecutor process inside gRPC server dies without throwing any errors

I have a gRPC server which uses ProcessPoolExecutor with max_workers=1 to run a script. Here is why i had to use a ProcessPoolExecutor inside run_brain_application with max_workers=1. Below is the simplified version of the server script. I've put…
James
  • 113
  • 4
7
votes
2 answers

Python concurrent futures ProcessPoolExecutor and global variables: works on Linux, error on MacOS

The code example below runs as I thought it should on two Linux machines: using Python 3.6.8 on a large CentOS-based server running Red Hat 4.8.5-39 kernel, and using Python 3.7.3 on my MX-based box running Debian 8.3.0-6 kernel). $ python3…
FSch
  • 101
  • 1
  • 5
7
votes
2 answers

Python concurrent.futures Error in atexit._run_exitfuncs: OSError: handle is closed only running in Visual studio Debugging Mode

I trying to implement the concurrent.futures have been having an issue, I generated a code that when executed in: Python Terminal Enters in some infinite loop in my last loop and do not finish the code, this code below do finish, however…
dtkx
  • 105
  • 1
  • 8
7
votes
2 answers

How can I get the arguments I sent to ThreadPoolExecutor when iterating through future results?

I use a ThreadPoolExecutor to quickly check a list of proxies to see which ones are dead or alive. with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: futures = [] for proxy in proxies: future =…
nathan
  • 73
  • 1
  • 3
7
votes
1 answer

Combining asyncio with a multi-worker ProcessPoolExecutor and for async

My question is very similar to Combining asyncio with a multi-worker ProcessPoolExecutor - however a slight change (I believe it's the async for) makes the excellent answers there unusuable for me. I am trying the following MWE: import…
adrug
  • 101
  • 3
  • 8
7
votes
0 answers

Use tqdm with multiprocessing for multiple progress bars

I want to monitor progress across multiple workers which are different processes. For each subprocess I have its own progress bar but it doest work properly with ProcessPoolExecutor executor. def main(): with…
Most Wanted
  • 6,254
  • 5
  • 53
  • 70
7
votes
2 answers

Flask - job not running as a background process

I am trying to run a Flask app which consists of: Yielding API requests on the fly Uploading each request to a SQLalchemy database Run jobs 1 and 2 as a background process For that I have the following code: import concurrent.futures import…
8-Bit Borges
  • 9,643
  • 29
  • 101
  • 198
7
votes
2 answers

How to give different names to ThreadPoolExecutor threads in Python

I have the below code for creating threads and running them. from concurrent.futures import ThreadPoolExecutor import threading def task(n): result = 0 i = 0 for i in range(n): result = result + i print("I:…
Subhayan Bhattacharya
  • 5,407
  • 7
  • 42
  • 60
7
votes
2 answers

Why does Future.onSuccess require a partial function

I am trying to chain together some basic functions using Futures returned from a slick action and I'm hitting some pretty trivial stumbling blocks. Both the andThen and onSuccess methods require a PartialFunction passed as a parameter. My…
GoldenFish
  • 217
  • 2
  • 9
7
votes
1 answer

Scala Nested Futures

I have a couple of futures. campaignFuture returns a List[BigInt] and I want to be able to call the second future profileFuture for each of the values in the list returned from the first one. The second future can only be called when the first one…
Anand
  • 1,791
  • 5
  • 23
  • 41
6
votes
1 answer

Deadlock in concurrent.futures code

I've been trying to parallelise some code using concurrent.futures.ProcessPoolExecutor but have kept having strange deadlocks that don't occur with ThreadPoolExecutor. A minimal example: from concurrent import futures def test(): pass with…
James
  • 3,191
  • 1
  • 23
  • 39
6
votes
1 answer

Nested Future.sequence executes included Futures sequentially

I have a future(doFour) that is executed and results passed to a flatmap. Inside the flatmap I execute two more future(doOne and doTwo) functions expecting them to run on parallel but I see they are running sequentially (2.13). Scastie Why are doOne…
nirfri
  • 63
  • 1
  • 2
1 2 3
…
61 62