Can I use map / imap / imap_unordered with functions with no arguments?

Question

Sometimes I need to use multiprocessing with functions with no arguments. I wish I could do something like:

from multiprocessing import Pool

def f():  # no argument
    return 1

# TypeError: f() takes no arguments (1 given)
print Pool(2).map(f, range(10))

I could do Process(target=f, args=()), but I prefer the syntax of map / imap / imap_unordered. Is there a way to do that?

Would you be willing to redefine `f` to take one argument and ignore it? — inspectorG4dget, Dec 29 '14 at 13:30
I have a feeling this is an [XY problem](http://meta.stackexchange.com/questions/66377/what-is-the-xy-problem). So, let's take a step back: what are you actually trying to do, for which you are trying to use this function? — inspectorG4dget, Dec 29 '14 at 13:43
The semantics of `map` imply that you're mapping a function *to a sequence of inputs*, so that's what you get. Whether you decide to ignore that argument, or create a wrapper function that's up to you. — Lukas Graf, Dec 29 '14 at 13:48
@inspectorG4dget: one example is to stress test a database or an API. I need to spawn N identical processes which will send requests repeatedly. — usual me, Dec 29 '14 at 13:54
In that case, please edit your post to show a representative example of exactly what you're trying to do, so that we can give you an answer that is hopefully both meaningful as well as to your liking — inspectorG4dget, Dec 29 '14 at 14:12
@inspectorG4dget So do you have another way of doing this that's not an XY problem? — endolith, Sep 06 '19 at 23:23

score 17 · Answer 1 · answered Feb 23 '19 at 12:50

17

You can use pool.starmap() instead of .map() like so:

from multiprocessing import Pool

def f():  # no argument
    return 1

print Pool(2).starmap(f, [() for _ in range(10)])

starmap will pass all elements of the given iterables as arguments to f. The iterables should be empty in your case.

answered Feb 23 '19 at 12:50

LearnOPhile

1,391
13
16

This can be rewritten as: `print(Pool(2).starmap(f, [[]] * 10))`. – young_souvlaki Jun 29 '22 at 22:01

thefourtheye · Accepted Answer · 2014-12-29T13:47:12.040

map function's first argument should be a function and it should accept one argument. It is mandatory because, the iterable passed as the second argument will be iterated and the values will be passed to the function one by one in each iteration.

So, your best bet is to redefine f to accept one argument and ignore it, or write a wrapper function with one argument, ignore the argument and return the return value of f, like this

from multiprocessing import Pool

def f():  # no argument
    return 1

def throw_away_function(_):
    return f()

print(Pool(2).map(throw_away_function, range(10)))
# [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

You cannot use lamdba functions with pools because they are not picklable.

it does not have to be a function accepting one argument only. you can use partial if you want to pass more arguments. — naivepredictor, Jun 17 '19 at 10:24

score 3 · Answer 3 · answered Dec 29 '14 at 14:49

3

Is there anything wrong with using Pool.apply_async?

with multiprocessing.Pool() as pool:
    future_results = [pool.apply_async(f) for i in range(n)]
    results = [f.get() for f in future_results]

answered Dec 29 '14 at 14:49

Dunes

37,291
7
81
97

1

This throws an error (Python 2.7): with multiprocessing.Pool() as pool: AttributeError: __exit__ – postoronnim Nov 13 '17 at 02:54

score 0 · Answer 4 · answered May 13 '22 at 19:21

0

For using the method pool, from library multiprocessing, you should provide an argument. Thus, you should use another method like Process from multiprocessing:

if __name__ == '__main__':
    p = Process(target=f)
    p.start()
    p.join()

answered May 13 '22 at 19:21

Sara

419
1
6
14

Can I use map / imap / imap_unordered with functions with no arguments?

4 Answers4

Linked