Questions tagged [starmap]
46 questions
1
vote
0 answers
starmap, starmap_async python
I want to translate a huge matlab model to python. Therefor I need to work on the key functions first. One key function handles parallel processing. Basically, a matrix with parameters is the input, in which every row represents the parameters for…
Lathano
- 23
- 2
1
vote
1 answer
Multiprocessing for creating objects + calling functions using starmap() Python
I would like to create objects of class Training and create multiple processes which call the print() function.
I have a class Training:
class Training():
def __init__(self, param1, param2):
self.param1 = param1
self.param2 =…
Asmita Poddar
- 524
- 1
- 11
- 27
1
vote
0 answers
Is there a way to run a function in parallel that takes ndarray as an argument?
I have a function,
def noflux(j,i,cells,rfu,lfu,ufu,dfu):
that takes numpy arrays as input and I would like to run this function twice in parallel using the multiprocessing package, i.e.
noflux(j,i,cells,rfu,lfu,ufu,dfu):
…
Conor McCann
- 11
- 2
1
vote
0 answers
Python multiprocessing pool within Class method
Context
I am trying to use multiprocessing, specifically Pool().starmap(), within a specific method of a class. Let's name the file with the code for the Quant class model.py. I'd like to be able to import Quant, declare an instance of it, myobj,…
Seabird86
- 11
- 1
1
vote
0 answers
Mutliprocessing.starmap : filtering
I'm using multiprocessing's Pool().starmap() function to run a type of cross-product computation, which is symmetrical, and i'm doing the same computation twice. How can i fix that to launch only half the processes ? What is taking me the more time…
lrnv
- 1,038
- 8
- 19
1
vote
0 answers
How can I exit a starmap_async process running in an multiprocessing pool?
I am having a lot of data (more than one million) and need to do some calculation on it that finds a certain value out of the millions. This is time consuming. To speed up the process I am trying to use all cores of the CPU. To do this, I am using…
hh221165
- 41
- 3
1
vote
4 answers
How to write a starmap-like function without using the builtin map or starmap?
This is the task:
Make a function my_map_k that as arguments takes a function f and k
lists L1,...,Lk, for an arbitrary k ≥ 1, and returns the list
[f(L1[0],...,Lk[0]),...,f(L1[n-1],...,Lk[n-1])], where n is the length
of the shortest Li…
NicklasS
- 35
- 6
1
vote
0 answers
How to Use Multiprocessing Over Dataframe
First off, thanks to anyone that responds. SO has saved so much time in my programming projects. I appreciate it!
My code iterates through a huge dataframe. Here is an example code:
#len(df)=1,000,000
for i in range(1,len(df))
…
Tanner Clark
- 631
- 1
- 8
- 19
0
votes
1 answer
Python: How to get the unordered result from starmap_async
So the thing is I want to get the return value from the starmap_async. However if I use the .get() to get the list of the result, it is little slow and same as starmap. How do I change the following sample code?
full_res = []
def…
master_zhen
- 37
- 1
- 4
0
votes
2 answers
Repeatedly process big list of images with changing parameters using multiple cores in python
I have a big list of images list_img, say 20k that I need to process multiple times with changing arguments out of a list params = [arg1, arg2, ...]. Ideally, I want to use multiple processes to do so. But I need all processes to first use arg1 and…
RaJa
- 1,471
- 13
- 17
0
votes
0 answers
Assertion error when using starmap multiprocessing in Python
I am trying to run a large data set of about 150 MM rows of data through Starmap multiprocessing in Python and Pandas. The .csv file with the dataset is about 18 GB.
# Multiprocessing pool
pool =…
Sughosh Kulkarni
- 73
- 2
- 8
0
votes
0 answers
Python multiprocessing freezes code in Ubuntu
I am trying to run a piece of python code that calls the starmap function
result = self.pool.starmap(CURR_CODE.__some_func__, args)
I ran this on arch linux and had no issues.
However, when I ran this on ubuntu, the code freezes. I don't get any…
Arpith_BioE
- 1
- 1
0
votes
0 answers
Starmap - AttributeError: 'function' object has no attribute 'children'?
#Full traceback
Exception ignored in: Exception ignored in: Exception ignored in:
DaPi123
- 1
- 1
0
votes
1 answer
Multiprocessing starmap_async get Pool ID to terminate
Iam working multiple multiprocessing pool starmap_async which was working fine, but the main issue is, how i can kill only particular pool.
tkinter UI
When i select the Camp1 and run it was running,it select list related to the campaign and running…
Naveen
- 1
- 2
0
votes
1 answer
End of the execution too long using Pool.starmap
I'm executing a paralelized function using Pool.starmap function. The execution of the function it self only takes 6.5 minutes according to tqdm library but the program stays in execution for 20 min more until it finishes. The function is processing…
kithuto
- 455
- 2
- 11