Stack Overflow
Stack Overflow

Questions tagged [concurrent.futures]

concurrent.futures is a Python module which provides a high-level interface for asynchronously executing callables.

The concurrent.futures module aims to provide a simple interface for parallelizing operations in multi-threaded and multi-process Python applications. The module was added to the Python standard library in version 3.2, but a backport is available for Python 2.5+.

929 questions
4
votes
1 answer

How to ensure a timeout per each Future in an iterator of concurrent.futures?

The documentation around timeouts for concurrent.futures is very challenging to understand. In a simple case, I'd like to use a ProcessPoolExecutor by calling .submit in a loop that scans through a list of job functions. I want each of these Future…
ely
  • 74,674
  • 34
  • 147
  • 228
4
votes
1 answer

Python concurrent.futures: ProcessPoolExecutor fail to work

I'm trying to use the ProcessPoolExecutor method but it fails. Here is an example(calculation of the big divider of two numbers) of a failed use. I don't understand what the mistake is def gcd(pair): a, b = pair low = min(a, b) for i in…
Avraham
  • 61
  • 1
  • 4
4
votes
0 answers

How to use Python Concurrent Futures with decorators

I'm using a decorator for the thread pool executor: from functools import wraps from .bounded_pool_executor import BoundedThreadPoolExecutor _DEFAULT_POOL = BoundedThreadPoolExecutor(max_workers=5) def threadpool(f, executor=None): @wraps(f) …
loretoparisi
  • 15,724
  • 11
  • 102
  • 146
4
votes
0 answers

Spark scala GroupByCount operation using scala futures

I have a dataFrame with 50 columns and I wanted to do groupBycount on 10 columns in parallel mode. val parallelism = Columns.length val executor = Executors.newFixedThreadPool(parallelism) val ec: ExecutionContext =…
Learnis
  • 526
  • 5
  • 25
4
votes
1 answer

How to use asyncio with ProcessPoolExecutor

I am searching for huge number of addresses on web, I want to use both asyncio and ProcessPoolExecutor in my task to quickly search the addresses. async def main(): n_jobs = 3 addresses = [list of addresses] _addresses =…
Awaish Kumar
  • 537
  • 6
  • 22
4
votes
1 answer

How to access generator object from executor.map?

I have function that converts non numerical data in a dataframe to numerical. import numpy as np import pandas as pd from concurrent import futures def convert_to_num(df): do stuff return df I am wanting to use the futures library to speed up…
RustyShackleford
  • 3,462
  • 9
  • 40
  • 81
4
votes
1 answer

Function that multiprocesses another function

I'm performing analyses of time-series of simulations. Basically, it's doing the same tasks for every time steps. As there is a very high number of time steps, and as the analyze of each of them is independant, I wanted to create a function that can…
Liris
  • 1,399
  • 3
  • 11
  • 29
4
votes
1 answer

How to use Queue in concurrent.futures.ProcessPoolExecutor()?

Disclaimer: I'm new to Python in general. I have a small experience with Go, where implementing a queue using a channel is really easy. I want to know how can I implement a Queue with ProcessPoolExecutor in Python 3. I want my N number of process…
cankentcode
  • 460
  • 1
  • 5
  • 20
4
votes
1 answer

Tornado unexpected exception in Future after timeout

I have set up a dask cluster. I can access a web dashboard, but when I'm trying to connect to the scheduler: from dask.distributed import Client client = Client('192.168.0.10:8786') I get the following error: tornado.application - ERROR - Exception…
Vladyslav Moisieienkov
  • 4,118
  • 4
  • 25
  • 32
4
votes
2 answers

Flask, concurrent.futures and SQLAlchemy - No application found: work inside a view function or push an application context

Im building a Flask app which require a background process resulting in uploads to a SQLAlchemy database. Relevant snippets: from flask_sqlalchemy import SQLAlchemy import concurrent.futures import queue from models import Upload_Tracks app =…
user7830303
4
votes
1 answer

Is it just me or something is seriously wrong with new Python futures module on windows

I am on windows XP and I have problems with new Python 3.2 futures module. It seems I am unable to get ProcessPoolExecutor to work. Session example: Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit (Intel)] on win32 Type "help",…
Piotr Lopusiewicz
  • 2,514
  • 2
  • 27
  • 38
4
votes
1 answer

How to use concurrent.futures in Python

Im struggling to get multithreading working in Python. I have i function which i want to execute on 5 threads based on a parameter. I also needs 2 parameters that are the same for every thread. This is what i have: from concurrent.futures import…
KnakworstKoning
  • 361
  • 3
  • 6
  • 18
4
votes
1 answer

How to wrap custom future to use with asyncio in Python?

There is a lot of libraries that use their custom version of Future. kafka and s3transfer are just two examples: all their custom future-like classes have object as the superclass. Not surprisingly, you cannot directly call asyncio.wrap_future() on…
kirill.shirokov
  • 43
  • 1
  • 4
4
votes
1 answer

How to access the data inside of Gathering Future finished result

Been trying to figure out how I can access my data after I return it. This is a sample of x if I print it: <_GatheringFuture finished result=[[{'ip': '173.245.203.70'}], [{'ip': '196.52.2.82'}], [{'ip': '69.161.4.1'}], [{'ip':'73.180.140.205'}],…
antfuentes87
  • 849
  • 3
  • 17
  • 34
4
votes
4 answers

Python doctest hangs using ProcessPoolExecutor

This code runs fine under regular CPython 3.5: import concurrent.futures def job(text): print(text) with concurrent.futures.ProcessPoolExecutor(1) as pool: pool.submit(job, "hello") But if you run it as python -m doctest myfile.py, it…
John Zwinck
  • 239,568
  • 38
  • 324
  • 436
1 2 3
…
61 62