Questions tagged [starmap]
46 questions
0
votes
1 answer
Python - pool.starmap() running much slower than pool.map()
To clarify first of all, I'm NOT asking why map in multiprocessing is slow.
I had code working just fine using pool.map(). But, in developing it (and to make it more generic), I needed to use pool.starmap() to pass 2 arguments instead of one.
I'm…
Kav
- 25
- 4
0
votes
1 answer
Python starmap got multiple values for argument
I hava a multiprocess program whcich using starmap. But when I run it, a TypeError occured. While normally when I run it in map.
def process_single_image(img_path, target_dir="", func=None, severity=1):
print(target_dir)
...
pool.starmap(
…
Whisht
- 681
- 2
- 6
- 20
0
votes
0 answers
Array inside object becomes empty, despite having contents when passed in using multiprocessing
Im new to python and having trouble passing in an object in a function.
Basically, I'm trying to read a large file over 1.4B lines.
I am passing in an object that contains information on the file. One of these is a very large array containing the…
Bit Scorpion
- 11
- 3
0
votes
0 answers
Understanding Blocking in starmap - Python Multiprocessing
In the python docs, it says that starmap blocks until the result is ready.
Does this mean that we can safely update a variable in main process by the results of child processes like this ?
from multiprocessing import Pool, cpu_count
from…
Saurabh Verma
- 6,328
- 12
- 52
- 84
0
votes
1 answer
Optimising python3 multiprocessing when each process requires multiple threads
I have a list with 12 items (each of which is a sample), and I need to analyse each of these samples using a function. The function is a wrapper for an external pipeline which needs four threads to run. Some pseudocode:
def my_wrapper(sample,…
frustrated_bioinformatician
- 353
- 2
- 10
0
votes
1 answer
How to correctly use Python multiprocessing for a function with signature f(str, str, str, list)?
Given the following code:
ids = [[1,2,3], [4,5], [6,7,8,9]]
it = [('a','b', 'c', ids[i]) for i in range(len(ids))]
# [('a', 'b', 'c', [1, 2, 3]), ('a', 'b', 'c', [4, 5]), ('a', 'b', 'c', [6, 7, 8, 9])]
p = multiprocessing.Pool(2)
j = p.starmap(f,…
Emmett
- 61
- 1
- 5
0
votes
0 answers
Code never enters 'with Pool(processes=4) as pool: block
New to multiprocessing in python and just trying to understand why my code won't go inside the 'with Pool(processes=4) as pool: block. Here is a simple code (I removed some details because they are not necessary).
import os
import psutil
import…
RforResearch
- 401
- 8
- 16
0
votes
1 answer
Passing in a dictionary as one argument of many using pool.starmap()
I'm trying to use pool.starmap() to make many concurrent post requests and URL retrievals. The function that I've built, let's call it "download", takes 4 arguments, 1 of them being a JSON payload. This function downloads a file and returns the…
itsrainingschmen
- 1
- 2
0
votes
2 answers
Python multiprocessing pool.map within an apply loop causing resets and strange behavior
I am experiencing some really weird behavior with pool.starmap in the context of a groupby apply function. Without getting the specifics, my code is something like this:
def groupby_apply_func(input_df):
print('Processing group # x (I determine…
trance_dude
- 91
- 1
- 7
0
votes
3 answers
Cannot pass class object to starmap in Python?
I am have a file and I want to process it in a parallelized manner using Python's multiprocessing class. My current code is:
class rand:
def __init__(self):
self.rando = "world"
def do_work2(obj, line):
return line + obj.rando
if…
R_Moose
- 103
- 9
0
votes
1 answer
multiprocessing: maxtasksperchild and chunksize conflict?
I am using the multiprocessing module in Python 3.7. My code is not working as expected (see this question here). Someone suggested to set maxtasksperchild, which I set to 1. Then, while reading the documentation, I figured that it was best to set…
YamiOmar88
- 1,336
- 1
- 8
- 20
0
votes
0 answers
Sequence of outputs by using pool starmap for parallel threads
I am using pool.starmap for parallel running of my function as follows.
def func_parallel(arg_1):
## Compute two ouputs and return those two outputs
## output_1=function_1(arg1)
## output_2=function_2(arg1)
return output_1,…
Stupid420
- 1,347
- 3
- 19
- 44
0
votes
1 answer
When using python's starmap, why is the location of a variable passed in using a closure different from that passed in using partial?
When I run the following code I see that variables z and w inside the function have different locations, but I don't understand why this is the case.
import multiprocessing
from functools import partial
class SomeClass:
pass
w =…
mm_857
- 171
- 11
0
votes
4 answers
Multiprocessing in python compatible from 3 to 2.7
I have the following code runnning smoothly in Python 3, and I can't convert it to Python 2.7.
from multiprocessing import *
def func(path, filename, path_2):
#Things to do
for i in range(0,len(chr_names)): #len(chr_names) = 24
…
D Grigor
- 1
- 3
0
votes
1 answer
Making parallel code work in python 2.7 and 3.6
I have some code in python 3.6 which is like this:
from multiprocessing import Pool
with Pool(processes=4) as p:
p.starmap(parallel_function, list(dict_variables.items()))
Here dict_variables looks like this:
[('aa', ['ab', 'ab', 'ad']),…
user308827
- 21,227
- 87
- 254
- 417