Stack Overflow
Stack Overflow

Questions tagged [concurrent.futures]

concurrent.futures is a Python module which provides a high-level interface for asynchronously executing callables.

The concurrent.futures module aims to provide a simple interface for parallelizing operations in multi-threaded and multi-process Python applications. The module was added to the Python standard library in version 3.2, but a backport is available for Python 2.5+.

929 questions
5
votes
2 answers

Why doesn't concurrent.futures make a copy of arguments?

My understanding was that concurrent.futures relied on pickling arguments to get them running in different processes (or threads). Shouldn't pickling create a copy of the argument? On Linux it does not seem to be doing so, i.e., I have to explicitly…
ariddell
  • 3,413
  • 34
  • 32
4
votes
2 answers

Python: ProcessPoolExecutor vs ThreadPoolExecutor

I have the following function that randomly shuffle the values of one column of the dataframe and use RandomForestClassifier on the overall dataframe including that column that is being randomly shuffled to get the accuracy score. And I would like…
user1769197
  • 2,132
  • 5
  • 18
  • 32
4
votes
1 answer

kill a future if program stops

I have a ThreadPoolExecutor in my programs which submit()s a task. However, when I end my program, the script "freezes". It seems like the thread is not ended correctly. Is there a solution for this? example: from concurrent.futures import…
DasSaffe
  • 2,080
  • 1
  • 28
  • 67
4
votes
0 answers

python ThreadPoolExecutor memory leak issues

I'm trying to debug a memory leak in my application, and I think I managed to reduce it to this minimal example: from typing import Deque import gc import os import psutil from concurrent.futures import ThreadPoolExecutor process =…
bentheiii
  • 451
  • 3
  • 15
4
votes
1 answer

Run multiple async loops in separate processes within a main async app

Ok so this is a bit convoluted but I have a async class with a lot of async code. I wish to parallelize a task inside that class and I want to spawn multiple processes to run a blocking task and also within each of this processes I want to create an…
KZiovas
  • 3,491
  • 3
  • 26
  • 47
4
votes
1 answer

Identify current thread in concurrent.futures.ThreadPoolExecutor

the following code has 5 workers .... each opens its own worker_task() with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor: future_to_url = {executor.submit(worker_task, command_, site_): site_ for site_ in URLS} for…
Rhys
  • 4,926
  • 14
  • 41
  • 64
4
votes
0 answers

ProcessPoolExecutor deadlocks with large number of tasks

I'm trying to run multiple identical short tasks with ProcessPoolExecutor on 256 cores Ubuntu Linux machine. The tasks are independent and don't share any resources. I initialize ProcessPoolExector with 252 cores, and submit it a large number of…
user107511
  • 772
  • 3
  • 23
4
votes
1 answer

How do I make Python respect iterable fields when multiprocessing?

Apologies if this is a dumb question, but I've not found an elegant workaround for this issue yet. Basically, when using the concurent.futures module, non-static methods of classes look like they should work fine, I didn't see anything in the docs…
Skullsploder
  • 43
  • 3
4
votes
0 answers

Reduce number of calls for the {method 'acquire' of '_thread.lock' objects} python

Hi There i'm struggling with my I/O bound app to make it fast enough for potential users im fetching an X number of urls say 10 for example, using MULTI THREADING with 1 thread for each URL but that takes too long i've ran Cprofile on my code and…
Driss ait
  • 66
  • 3
4
votes
1 answer

Max Workers in `multiprocessing` vs `concurrent.futures`

In Python 3.8, concurrent.futures.ProcessPoolExecutor has been updated to limit the max number of workers (processes) able to be used on Windows to 61. For the reasons why, see this and this, but to my understanding: On Windows, multiprocessing…
adam.hendry
  • 4,458
  • 5
  • 24
  • 51
4
votes
1 answer

Python use concurrent.futures in streamlit got "missing ReportContext"

I got many warnings in console when I used concurrent.futures in my streamlit app. Some of warnings : 2020-10-28 15:43:59.338 Thread 'ThreadPoolExecutor-1_11': missing ReportContext 2020-10-28 15:43:59.338 Thread 'ThreadPoolExecutor-1_8': missing…
onon99buynoodle
  • 141
  • 8
4
votes
0 answers

Passing a Queue with concurrent.futures regardless of executor type

Working up from threads to processes, I have switched to concurrent.futures, and would like to gain/retain flexibility in switching between a ThreadPoolExecutor and a ProcessPoolExecutor for various scenarios. However, despite the promise of a…
matanster
  • 15,072
  • 19
  • 88
  • 167
4
votes
2 answers

Equivalent of js then() in python?

In Typescript I'm used to writing async code like this: async function foo() // returns a promise return new Promise( resolve, reject) { resolve('Hello!'); }); async function bar() { s: string = await foo(); } async…
Josh Greifer
  • 3,151
  • 24
  • 25
4
votes
2 answers

Getting progress update from concurrent.futures

I would like to copy a file from a separate thread or process (whichever is faster), so as not to block the main thread. I'd also like to get an occasional progress update. With a "regular" worker thread, I can push the progress stat to a Queue, and…
Jay
  • 2,535
  • 3
  • 32
  • 44
4
votes
1 answer

Python: exec with ProcessPoolExecutor

from concurrent.futures import ProcessPoolExecutor import os import time def parInnerLoop(item): a = 2+item print(f'A. {a} Processing {os.getpid()} done on {item}\n') exec(open('mainWork.py').read()) print(f'D. {a} Processing…
Dr.PB
  • 959
  • 1
  • 13
  • 34
1 2 3
…
61 62