Stack Overflow
Stack Overflow

Questions tagged [concurrent.futures]

concurrent.futures is a Python module which provides a high-level interface for asynchronously executing callables.

The concurrent.futures module aims to provide a simple interface for parallelizing operations in multi-threaded and multi-process Python applications. The module was added to the Python standard library in version 3.2, but a backport is available for Python 2.5+.

929 questions
24
votes
2 answers

Checking up on a `concurrent.futures.ThreadPoolExecutor`

I've got a live concurrent.futures.ThreadPoolExecutor. I want to check its status. I want to know how many threads there are, how many are handling tasks and which tasks, how many are free, and which tasks are in the queue. How can I find out these…
Ram Rachum
  • 84,019
  • 84
  • 236
  • 374
24
votes
2 answers

What's the difference between python's multiprocessing and concurrent.futures?

A simple way of implementing multiprocessing in python is from multiprocessing import Pool def calculate(number): return number if __name__ == '__main__': pool = Pool() result = pool.map(calculate, range(4)) An alternative…
David Zwicker
  • 23,581
  • 6
  • 62
  • 77
19
votes
3 answers

How to pass a function with more than one argument to python concurrent.futures.ProcessPoolExecutor.map()?

I would like concurrent.futures.ProcessPoolExecutor.map() to call a function consisting of 2 or more arguments. In the example below, I have resorted to using a lambda function and defining ref as an array of equal size to numberlist with an…
Sun Bear
  • 7,594
  • 11
  • 56
  • 102
19
votes
2 answers

Why does java.util.concurrent.RunnableFuture have a run() method?

As I was going through JDK 7, I found that java.util.concurrent.RunnableFuture has a run method. I wonder what the significance of duplicating the same run method signature in the interface is when it already extends Runnable. package…
Maas
  • 1,317
  • 9
  • 20
19
votes
3 answers

Multiprocessing Share Unserializable Objects Between Processes

There are three questions as possible duplicates (but too specific): How to properly set up multiprocessing proxy objects for objects that already exist Share object with process (multiprocess) Can I use a ProcessPoolExecutor from within a…
User
  • 14,131
  • 2
  • 40
  • 59
18
votes
2 answers

How to detect exceptions in concurrent.futures in Python3?

I have just moved on to python3 as a result of its concurrent futures module. I was wondering if I could get it to detect errors. I want to use concurrent futures to parallel program, if there are more efficient modules please let me know. I do not…
user5327424
18
votes
1 answer

Python ThreadPoolExecutor - is the callback guaranteed to run in the same thread as submitted func?

In the ThreadPoolExecutor (TPE), is the callback always guaranteed to run in the same thread as the submitted function? For example, I tested this with the following code. I ran it many times and it seemed like func and callback always ran in the…
Praveen Gollakota
  • 37,112
  • 11
  • 62
  • 61
18
votes
3 answers

A ThreadPoolExecutor inside a ProcessPoolExecutor

I am new to the futures module and have a task that could benefit from parallelization; but I don't seem to be able to figure out exactly how to setup the function for a thread and the function for a process. I'd appreciate any help anyone can shed…
inspectorG4dget
  • 110,290
  • 27
  • 149
  • 241
18
votes
1 answer

What are the advantages of concurrent.futures over multiprocessing in Python?

I'm writing an app in Python and I need to run some tasks simultaneously. The module multiprocessing offers the class Process and the concurrent.futures module has the class ProcessPoolExecutor. Both seem to use multiple processes to execute their…
Daniel Jonsson
  • 3,261
  • 5
  • 45
  • 66
17
votes
2 answers

concurrent.futures.ThreadPoolExecutor.map(): timeout not working

import concurrent.futures import time def process_one(i): try: print("dealing with {}".format(i)) …
Hao Wang
  • 499
  • 5
  • 19
17
votes
2 answers

Python parallel execution with selenium

I'm confused about parallel execution in python using selenium. There seems to be a few ways to go about it, but some seem out of date. There's a python module called python-wd-parallel which seems to have some functionality to do this, but it's…
Ke.
  • 2,484
  • 8
  • 40
  • 78
16
votes
1 answer

store results ThreadPoolExecutor

I am fairly new to parallel processing with "concurrent.futures" and I am testing some simple experiments. The code I have written seems to work, but I am not sure how to store the results. I have tried to create a list ("futures") and append the…
Mike
  • 375
  • 1
  • 4
  • 14
16
votes
2 answers

multiprocessing queue full

I'm using concurrent.futures to implement multiprocessing. I am getting a queue.Full error, which is odd because I am only assigning 10 jobs. A_list = [np.random.rand(2000, 2000) for i in range(10)] with ProcessPoolExecutor() as pool: …
rohanp
  • 610
  • 1
  • 8
  • 20
15
votes
3 answers

Can concurrent.futures.Future be converted to asyncio.Future?

I'm practicing asyncio after writing multithreaded code many years. Noticed something which i find it strange. Both in asyncio and in concurrent there is a Future object. from asyncio import Future from concurrent.futures import Future Guess each…
Aaron_ab
  • 3,450
  • 3
  • 28
  • 42
14
votes
1 answer

Correctly using loop.create_future

I was reading the Python documentation and PyMotW book trying to learn Async/Await, Futures, and Tasks. Coroutines and Tasks documentation: Normally there is no need to create Future objects at the application level code. From the future…
user10597450
1
2
3
…
61 62