Stack Overflow
Stack Overflow

Questions tagged [concurrent.futures]

concurrent.futures is a Python module which provides a high-level interface for asynchronously executing callables.

The concurrent.futures module aims to provide a simple interface for parallelizing operations in multi-threaded and multi-process Python applications. The module was added to the Python standard library in version 3.2, but a backport is available for Python 2.5+.

929 questions
12
votes
2 answers

Why I cannot use python module concurrent.futures in class method?

I want to make my class method runs parallel, but it only produces some kind of error that I can not solve. My code is: import concurrent.futures as futures samples = ['asfd', 'zxcv', 'asf', 'qwer'] class test: def __init__(self, samples): …
user2028191
  • 121
  • 1
  • 1
  • 3
11
votes
2 answers

Python - Properly Kill/Exit Futures Thread?

I was previously using the threading.Thread module. Now I'm using concurrent.futures -> ThreadPoolExecutor. Previously, I was using the following code to exit/kill/finish a thread: def terminate_thread(thread): """Terminates a python thread from…
xendi
  • 2,332
  • 5
  • 40
  • 64
11
votes
1 answer

Is it possible to pass Python Future objects between processes?

Based on my experiments I'm guessing the answer to this is no. But perhaps it might be possible with some changes to the futures module. I would like to submit a worker that itself creates an executor and submits work. I want to return that second…
Erotemic
  • 4,806
  • 4
  • 39
  • 80
10
votes
1 answer

Future raising TypeError after wait

I'm trying to optimise an expensive operation in some existing code using parallel processing. I've used concurrent.futures to do so in the past but only when they didn’t return anything. This time I want to marshall the results, but when printing…
Robin Whittleton
  • 6,159
  • 4
  • 40
  • 67
10
votes
2 answers

populate numpy array through concurrent.futures multiprocessing

I'm seeking to populate a large numpy array using multiprocessing. I've worked through the concurrent futures examples in the documentation but haven't obtained enough of an understanding to modify the usage. Here's a simplified version of what I'd…
zazizoma
  • 437
  • 1
  • 7
  • 18
10
votes
4 answers

concurrent.futures.ThreadPoolExecutor.map is slower than a for loop

I am playing with concurrent.futures.ThreadPoolExecutor to see if I can squeeze more work out of my quad-core processor (with 8 logical cores). So I wrote the following code: from concurrent import futures def square(n): return n**2 def…
inspectorG4dget
  • 110,290
  • 27
  • 149
  • 241
9
votes
4 answers

Python ThreadPoolExecutor terminate all threads

I am running a piece of python code in which multiple threads are run through threadpool executor. Each thread is supposed to perform a task (fetch a webpage for example). What I want to be able to do is to terminate all threads, even if one of the…
Khizar Amin
  • 198
  • 1
  • 2
  • 12
9
votes
6 answers

Multithreading inside Multiprocessing in Python

I am using concurrent.futures module to do multiprocessing and multithreading. I am running it on a 8 core machine with 16GB RAM, intel i7 8th Gen processor. I tried this on Python 3.7.2 and even on Python 3.8.2 import concurrent.futures import…
learner
  • 828
  • 2
  • 19
  • 36
9
votes
1 answer

CompletableFuture is not getting executed. If I use the ExecutorService pool it works as expected but not with the common ForkJoinPool

I am trying to run the following class its getting terminated without executing the CompletableFuture. public class ThenApplyExample { public static void main(String[] args) throws Exception { //ExecutorService es =…
Piyush Kumar
  • 119
  • 1
  • 7
9
votes
5 answers

How to break time.sleep() in a python concurrent.futures

I am playing around with concurrent.futures. Currently my future calls time.sleep(secs). It seems that Future.cancel() does less than I thought. If the future is already executing, then time.sleep() does not get cancel by it. The same for the…
guettli
  • 25,042
  • 81
  • 346
  • 663
9
votes
4 answers

In Futures.transform, what is the difference between using a Function and an AsyncFunction

I know that the apply method of Function returns an object synchronously, and the apply of AsyncFunction runs asynchronously and returns a Future. Can you give me an example of when to prefer what. One code snippet that I saw looked something like…
SherinThomas
  • 1,881
  • 4
  • 16
  • 20
9
votes
2 answers

How to use queue with concurrent future ThreadPoolExecutor in python 3?

I am using simple threading modules to do concurrent jobs. Now I would like to take advantages of concurrent futures modules. Can some put me a example of using a queue with concurrent library? I am getting TypeError: 'Queue' object is not…
user2433024
  • 109
  • 1
  • 1
  • 4
8
votes
1 answer

ThreadPoolExecutor with Context Manager "cannot schedule new futures after shutdown"

I'm creating a thread manager class that handles executing tasks as threads and passing the results to the next process step. The flow works properly upon the first execution of receiving a task, but the second execution fails with the following…
sadmicrowave
  • 39,964
  • 34
  • 108
  • 180
8
votes
0 answers

Python 3.8 concurrent.futures "OSError: handle is closed"

I have Python 3.8 and cannot upgrade (I have dependencies that only work with Python 3.8), so this SO post does not answer my question. How do I prevent the OSError: handle is closed error? This happens when returning a result to the main thread.…
adam.hendry
  • 4,458
  • 5
  • 24
  • 51
8
votes
2 answers

Any concurrent.futures timeout that actually works?

Tried to write a process-based timeout (sync) on the cheap, like this: from concurrent.futures import ProcessPoolExecutor def call_with_timeout(func, *args, timeout=3): with ProcessPoolExecutor(max_workers=1) as pool: future =…
wim
  • 338,267
  • 99
  • 616
  • 750
1 2 3
…
61 62