Stack Overflow
Stack Overflow

Questions tagged [concurrent.futures]

concurrent.futures is a Python module which provides a high-level interface for asynchronously executing callables.

The concurrent.futures module aims to provide a simple interface for parallelizing operations in multi-threaded and multi-process Python applications. The module was added to the Python standard library in version 3.2, but a backport is available for Python 2.5+.

929 questions
6
votes
2 answers

Shared variable in concurrent.futures.ProcessPoolExecutor() python

I want to use parallel to update global variable using module concurrent.futures in python It turned out that using ThreadPoolExecutor can update my global variable but the CPU did not use all their potential (always at 5-10%), which is so slow and…
Hiếu Ngô
  • 79
  • 1
  • 1
  • 5
6
votes
1 answer

Python multithreading/multiprocessing very slow with concurrent.futures

I am trying to use multithreading and/or multiprocessing to speed up my script somewhat. Essentially I have a list of 10,000 subnets I read in from CSV, that I want to convert into an IPv4 object and then store in an array. My base code is as…
Andrew Harris
  • 101
  • 1
  • 5
6
votes
2 answers

How to fix BrokenProcessPool: error for concurrent.futures ProcessPoolExecutor

Using concurrent.futures.ProcessPoolExecutor I am trying to run the first piece of code to execute the function "Calculate_Forex_Data_Derivatives(data,gride_spacing)" in parallel. When calling the results, executor_list[i].result(), I get…
ZachV
  • 71
  • 1
  • 1
  • 4
6
votes
1 answer

concurrent.futures.ThreadPoolExecutor doesn't print errors

I am trying to use concurrent.futures.ThreadPoolExecutor module to run a class method in parallel, the simplified version of my code is pretty much the following: class TestClass: def __init__(self, secondsToSleepFor): …
ela
  • 325
  • 2
  • 10
6
votes
4 answers

ThreadPoolExecutor, ProcessPoolExecutor and global variables

I am new to parallelization in general and concurrent.futures in particular. I want to benchmark my script and compare the differences between using threads and processes, but I found that I couldn't even get that running because when using…
Bram Vanroy
  • 27,032
  • 24
  • 137
  • 239
6
votes
1 answer

The use of ">>" in Pharo/Smalltalk

I am implementing futures in Pharo. I came across this website http://onsmalltalk.com/smalltalk-concurrency-playing-with-futures. I am following this example and trying to replicate it on Pharo. However, I get to this point the last step and I have…
Gakuo
  • 845
  • 6
  • 26
6
votes
0 answers

using pathos with ProcessPoolExecutor

its possible to use concurrent.futures (ProcessPoolExecutor) with pathos ? asking this because when i try to use: with concurrent.futures.ProcessPoolExecutor() as executor: ... executor.map() i will get: PicklingError: Can't pickle : attribute…
xampione
  • 71
  • 1
  • 6
6
votes
2 answers

Calling pyspark function asynchronously with concurrent.futures

I am trying to call python functions which use pyspark rdd objects methods and are time-consuming which blocks my application. I need to write it in an async fashion so that my app doesn't get blocked. Here is a miniature version of the actual thing…
DevanshBheda
  • 91
  • 1
  • 6
6
votes
0 answers

Is it possible to add new jobs dynamically using concurrent.futures.ThreadPoolExecutor?

The use case I have in mind is as follows: I would like to start a range of jobs with ThreadPoolExecutor and then when a job completes, I would like to add a new job to the queue. I would also like to know when the next job finishes and repeat the…
Alma Rahat
  • 305
  • 3
  • 14
6
votes
2 answers

Python typings and futures

I'm very glad to use typing module in Python 3. Also, I'm very glad to use asyncio instead of twisted, tornado and alternatives. My question is how to define result of a coroutine properly? Should we tell it's just a coroutine? Example 1: async def…
oblalex
  • 5,366
  • 2
  • 24
  • 25
6
votes
1 answer

How to pass keyword argument to function called by concurrent.futures map call

I have the following code: from concurrent.futures import ThreadPoolExecutor def spam(url, hello=None, params=None): print(url, hello, params) urls = [1, 2, 3, 4, 5] params = [(6, 7), 7, ('a', 1), 9, 'ab'] with ThreadPoolExecutor(5) as…
mart1n
  • 5,969
  • 5
  • 46
  • 83
6
votes
0 answers

Python: concurrent.futures: cancel not possible

This little snippets prints False import subprocess from concurrent import futures with futures.ThreadPoolExecutor(max_workers=1) as executor: future=executor.submit(subprocess.call, ['sleep', '2']) print(future.cancel()) According to the…
guettli
  • 25,042
  • 81
  • 346
  • 663
6
votes
3 answers

Concurrent.futures > works well in command line, not when compiled with pyinstaller or py2exe

I have a very simple script based on concurrent.futures that works well in command line (Python 2.7), but crashes when compiled with py2exe or Pyinstaller (the compiled program opens an increasing number of processes and eventually completely blocks…
Steph
  • 61
  • 3
6
votes
1 answer

What is the difference between using the method ThreadPoolExecutor.shutdown(wait=True), shutdown(wait=False) and without using this one?

I can't understand difference. Help me to watch this difference. And what about ProcessPoolExecutor, is his behavior the same? def func(task): do_something(task) tasks = [task for i in range(12)] executor =…
kvdm.dev
  • 141
  • 1
  • 1
  • 12
6
votes
1 answer

Timeout for each thread in ThreadPool in python

I am using Python 2.7. I am currently using ThreadPoolExecuter like this: params = [1,2,3,4,5,6,7,8,9,10] with concurrent.futures.ThreadPoolExecutor(5) as executor: result = list(executor.map(f, params)) The problem is that f sometimes runs for…
user302099
  • 635
  • 1
  • 5
  • 17
1 2 3
…
61 62