Stack Overflow
Stack Overflow

Questions tagged [concurrent.futures]

concurrent.futures is a Python module which provides a high-level interface for asynchronously executing callables.

The concurrent.futures module aims to provide a simple interface for parallelizing operations in multi-threaded and multi-process Python applications. The module was added to the Python standard library in version 3.2, but a backport is available for Python 2.5+.

929 questions
0
votes
1 answer

Multithreading: URL fetching

I tried to multiprocess a URL-fetching process since it would otherways take a massive amount of time to process the 300k urls I want to process. Somehow my code stops working after a random amount of time and I do not why. Can you help me? I…
inneb
  • 1,060
  • 1
  • 9
  • 20
0
votes
0 answers

concurrent.futures job deleting itself from a list -> IndexError in

I've encountered an error that I cannot explain while try to retrieve the results of futures submitted to process pool. I've stored the future objects in a list, and my best guess is that the future object reference is being deleted somehow so that…
Josh Albert
  • 1,064
  • 13
  • 16
0
votes
0 answers

Python's ThreadPoolExecutor is unable to load modules in threads running concurrently

I am executing a function using concurrent.futures.ThreadPoolExecutor() where each thread loads a module from site-packages. Here is how the code goes, def execFn(r): try: module = sys.modules['xxx.abc'] except KeyError: …
Mahalakshmi Suresh Kumar
  • 35
  • 8
0
votes
0 answers

Is it possible to make the callback be handled under the same subprocess?

from concurrent.futures import ProcessPoolExecutor import threading import os def task(): print('Task: in thread', threading.current_thread()) print('Task: in process', os.getpid()) def taskDone(fn): print('Callback: in thread',…
Menglong Li
  • 2,177
  • 14
  • 19
0
votes
1 answer

Klein app with deferred

I am exploring Klein and Deferred. In the following example I am trying to increment a number using a child process and return it via Future. I am able to receive the Future call back. The problem is that deferred object never calls the cb()…
Majeed Khan
  • 505
  • 7
  • 16
0
votes
1 answer

How to apply a function to mulitple columns of a pandas DataFrame in parallel

I have a pandas DataFrame with hundreds of thousands of rows, and I want to apply a time-consuming function on multiple columns of that DataFrame in parallel. I know how to apply the function serially. For example: import hashlib import pandas as…
ostrokach
  • 17,993
  • 11
  • 78
  • 90
0
votes
1 answer

how to map class and function in concurrent.futures.ProcessPoolExecute

I have done a task of concurrent in Python, as the following code def fun_x(a, b) : for x in range(10000) : x = x*x for y in range(10000) : y = y*y return a*x, b*y with futures.ProcessPoolExecutor() as…
Carl Zheng
  • 708
  • 2
  • 6
  • 20
0
votes
0 answers

Python: Using multiprocess to run a function from an external script

I am facing some issues while using concurrent.futures Multiprocess module. import sys import ConfigParser import os import concurrent.futures from Script_A import get_list, AnalyzeRow # rec_list = get_rec_list(MODE) with…
user8034399
  • 1
  • 3
0
votes
2 answers

Nested loops output to dict in parallel

I have two set of data: aDict = {'barcode1': [('barcode1', 184), ('barcode1_mut', 2)], 'barcode2': [('barcode2', 138)], 'barcode3': [('barcode3', 375)]} bList = [(('barcode1', 'mut1'), 184), (('barcode1_mut', 'mut2'), 2), (('barcode2', 'mut3'),…
Anton Ivankin
  • 47
  • 8
0
votes
1 answer

Python: How to get hung process to release control back in concurrent.futures pool?

I have a parallelized a large CPU-intensive data processing task using the concurrent.futures ProcessPoolExecutor method like shown below. with concurrent.futures.ProcessPoolExecutor(max_workers=workers) as executor: futures_ocr = ([ …
sfactor
  • 12,592
  • 32
  • 102
  • 152
0
votes
1 answer

How to pass HashMap from one Actor to another

I have a method getAggOutput() that returns a HashMap, in Actor A. def getAggOutput: HashMap[X, List[Y]] = { println("***** Inside getAggOutput, rMap is: " + rMap) return rMap } Also, in Actor A, the method is called that under the case…
user1317750
  • 51
  • 2
  • 8
0
votes
1 answer

pass more than one function to a callback

Given that : with concurrent.futures.ProcessPoolExecutor(max_workers=(2*multiprocessing.cpu_count()+1)) as executor: for netelement in DOC['code']['info']['dev']: job = executor.submit(bgp_communities.do_lookup, netelement) …
nskalis
  • 2,232
  • 8
  • 30
  • 49
0
votes
1 answer

python concurrent.futures getting result slows down

when I try getting the result, it feels like it goes back too not being threaded, but when I don't grab the result. it works like it has 10 threads, any idea why or how I can fix this? pool = ThreadPoolExecutor(max_workers=10) info =…
user6427229
0
votes
1 answer

getting controller return type error in playframework when trying to return Future[Result]

I have a pretty complex flow of my controller, this is how it goes using words: - get the request - translate it to my model case it succeed: insert some record to my db case it succeed: perform some api call …
JohnBigs
  • 2,691
  • 3
  • 31
  • 61
0
votes
1 answer

Scala: Recursion with future Stack overflow error

Consider below code snippet. Too many take() calls will result in stack overflow error. What is the good strategy to overcome such scenarios? def take(): Future[Int] = { val processResponseF: Future[Boolean] = performSomeAction() …
user3635344
  • 13
  • 4
1 2 3
…
61 62