Stack Overflow
Stack Overflow

Questions tagged [concurrent.futures]

concurrent.futures is a Python module which provides a high-level interface for asynchronously executing callables.

The concurrent.futures module aims to provide a simple interface for parallelizing operations in multi-threaded and multi-process Python applications. The module was added to the Python standard library in version 3.2, but a backport is available for Python 2.5+.

929 questions
5
votes
1 answer

Memory leak (?) in multithreaded python project

I have a small project of mine (please keep in mind that I'm just a python beginner). This project consists of few smaller .py files. First there is main.py that looks like this: from Controller import Controller import config as cfg if __name__ ==…
jlipinski
  • 152
  • 12
5
votes
3 answers

How to terminate Python's `ProcessPoolExecutor` when parent process dies?

Is there a way to make the processes in concurrent.futures.ProcessPoolExecutor terminate if the parent process terminates for any reason? Some details: I'm using ProcessPoolExecutor in a job that processes a lot of data. Sometimes I need to…
Rich
  • 12,068
  • 9
  • 62
  • 94
5
votes
2 answers

ThreadPoolExecutor with stateful workers

I'm working with a Backend class which spawns a subprocess to perform the CPU-bound work. I have no control over that class and basically the only way of interaction is to create an instance backend = Backend() and submit work via backend.run(data)…
a_guest
  • 34,165
  • 12
  • 64
  • 118
5
votes
1 answer

ThreadPoolExecutor KeyboardInterrupt

I've got the following code which uses a concurrent.futures.ThreadPoolExecutor to launch processes of another program in a metered way (no more than 30 at a time). I additionally want the ability to stop all work if I ctrl-C the python process. …
sheridp
  • 1,386
  • 1
  • 11
  • 24
5
votes
2 answers

malloc(): unsorted double linked list corrupted Aborted (core dumped) python

While trying to download files from google drive concurrently using concurrent.futures module the below script throwing malloc(): unsorted double linked list corrupted. files = [ {"id": "2131easd232", "name": "image1.jpg"}, {"id":…
mohammed wazeem
  • 1,310
  • 1
  • 10
  • 26
5
votes
2 answers

Seeding numpy.random's default_rng and SeedSequence objects for concurrent.futures.ProcessPoolExecutor

I am learning to set up the seed of NumPy ver 1.19 psuedo-random number generator for a Python 3.6 concurrent.futures.ProcessPoolExecutor analysis. After reading NumPy's documentation on Random sampling and Parallel Random Number Generation, I…
Sun Bear
  • 7,594
  • 11
  • 56
  • 102
5
votes
1 answer

ImportError: cannot import name 'ProcessPoolExecutor' from 'concurrent.futures.process'

I have a problem that I cannot figure out what happened: I was trying to follow a multiprocessing tutorial using the following code: import concurrent.futures import time start = time.perf_counter() def do_something(seconds): print(f'sleeping…
Jiang Xu
  • 91
  • 1
  • 10
5
votes
1 answer

How to apply executor.map to for loop without a function?

I have a list of xml and a for loop that flattens the xml into a pandas dataframe. The for loop works perfectly fine but is taking very long to flatten the xml, which is getting larger as time goes on. How do I wrap the below for-loop in…
RustyShackleford
  • 3,462
  • 9
  • 40
  • 81
5
votes
0 answers

Using Concurrent.Futures and safely writing to a SQLite file

I'm processing several thousand XML files, and wish to do it with use ProcessPoolExecutor from concurrent.futures. The sole documentation example assume that you wait for all the results before continuing (as do all concurrent.futures tutorials I've…
GIS-Jonathan
  • 4,347
  • 11
  • 31
  • 45
5
votes
1 answer

Python Process pool executor shutdown on signal

I use supervisor to run some script like this. so when supervisor is stopped or interrupt, i try to gracefully exit from the script. This is my current code import concurrent.futures import random import os import signal import sys executor =…
Anandan
  • 353
  • 3
  • 17
5
votes
1 answer

How to run generator code in parallel?

I have code like this: def generator(): while True: # do slow calculation yield x I would like to move the slow calculation to separate process(es). I'm working in python 3.6 so I have concurrent.futures.ProcessPoolExecutor. …
Alex I
  • 19,689
  • 9
  • 86
  • 158
5
votes
1 answer

Multithreaded HTTP GET requests slow down badly after ~900 downloads

I'm attempting to download around 3,000 files (each being maybe 3 MB in size) from Amazon S3 using requests_futures, but the download slows down badly after about 900, and actually starts to run slower than a basic for-loop. It doesn't appear that…
shadowtalker
  • 12,529
  • 3
  • 53
  • 96
5
votes
1 answer

What are the benefits of wrapping Callable/Runnable in a FutureTask?

What does FutureTask wrapper offer over simple Callable/Runnables? I've seen some people using futures that way, but I am not sure what it really adds to the game. Callable myComputation = () -> {return 0;}; FutureTask task = new…
Whimusical
  • 6,401
  • 11
  • 62
  • 105
5
votes
2 answers

How to print results of Python ThreadPoolExecutor.map immediately?

I am running a function for several sets of iterables, returning a list of all results as soon as all processes are finished. def fct(variable1, variable2): # do an operation that does not necessarily take the same amount of # time for…
AKG
  • 1,799
  • 4
  • 13
  • 23
5
votes
1 answer

Run python threads on multiple cores

I know that Python 2.7 does not allow one to run multiple threads on different cores, and you need to use the multiprocessing module in order to achieve some degree of concurrency. I was looking at the concurrent.futures module in Python 3.4. Does…
Diptesh Chatterjee
  • 365
  • 5
  • 14
1 2 3
…
61 62