how to handover *args to pathos' amap

Question

I have the following problem: I would like to use a amap from pathos.multiprocessing.

import pathos as pt

class Foo:

    def __init__(self):
        pass

    def f(self, a, b, c):
        return a + b + c

    def g(self):  
        pool = pt.multiprocessing.ProcessPool()
        def _h(a, b, c):
            k = self.f(a, b, c)
            return k/2
        result = [pool.amap(_h, (i, j, k)) for i in range(3) for j in range(5) for k in range(7)]
        return result

a = Foo()
b = a.g()
b[0].get()

It is necessary to have these two function f and g although I could do everything within f.

If I run this code I get that g expects 3 arguments but one was given.

TypeError: _h() takes exactly 3 arguments (1 given)

How can I fix this?

I don't have pathos and can't find its docs, but it seems like you could either rewrite `_h` to take a tuple or see if pathos has a `starmap` analogue. — user2357112, Oct 19 '17 at 16:55
...wait, you're passing a single tuple to amap, not a list of tuples. That seems completely wrong; something with `map` in the name should take an iterable to map the function over, not a single argument tuple. — user2357112, Oct 19 '17 at 16:57
@user2357112 thx for your comment. you can find pathos here: https://github.com/uqfoundation/pathos . about your second comment. even if I use a list of tuple it doesn't work — math, Oct 19 '17 at 17:21

georgexsh · Accepted Answer · 2017-10-21T16:51:03.010

because amap defined as:

149     def amap(self, f, *args, **kwds): # register a callback ?
...
152         return _pool.map_async(star(f), zip(*args)) # chunksize

there is an usage example in source code:

pool.amap(pow, [1,2,3,4], [5,6,7,8])

given l as your input:

l = [(i, j, k) for i in range(3) for j in range(5) for k in range(7)]

you could transpose your input:

results = pool.amap(_h, *map(list, zip(*l)))

or use a generator, which is supposed to be faster:

def getter(n):
    for e in l:
        yield e[n]
result = pool.amap(_h, *[getter(n) for n in range(3)])

or, use apipe api instead:

results = [pool.apipe(_h, l)]

of course, you could make input more suitable with its interface once you get the idea. but why not just use multiprocessing.pool.async_apply instead, its interface is exactly as you expected at first.

+1. many thanks for your answer. I will wait until I accept an answer and distribute the bounty to leave it open for others as well. One question: Is there a more efficient way to build inputs `l` ? — math, Oct 20 '17 at 14:30

score 0 · Answer 2 · answered Oct 19 '17 at 21:54

0

I know it looks strange, however I like the idea of passing data as iterable in pathos.

To achieve what you are looking for you must pass the elements as a separate iterable item. Here is an example:

def g(self):  
    pool = pt.multiprocessing.ProcessPool()
    def _h(a, b, c):
        k = self.f(a, b, c)
        return k/2
    result = [pool.amap(_h, (i,),( j,),(k,))) for i in range(3) for j in range(5) for k in range(7)]
    return result

Please notice the strange way of passing parameters here: pool.amap(_h, (i,),( j,),(k,))

You can get an idea why it is done so if you trace _pool.map_async(star(f), zip(*args)) call a bit.

In general, you might want to call your function multiple times with different parameters. Here is what I've done to demonstrate it:

def g(self):  
    pool = pt.multiprocessing.ProcessPool()
    def _h(a, b, c):
        print('a: {0} b: {1}. c: {2}'.format(a, b, c))
        k = self.f(a, b, c)
        return k/2
    return [pool.amap(_h, (1,4), (2,5), (3,6))]

Even when I called function once explicitly it got executed twice. Output:

a: 1 b: 2. c: 3
a: 4 b: 5. c: 6

Hope this helps.

answered Oct 19 '17 at 21:54

taras

3,579
3
26
27

I would be happy to know a reason :( – taras Oct 20 '17 at 05:40
+1 one. Thanks for your answer. I will keep the bounty open for a moment. But it looks like this is exactly what I was looking for – math Oct 20 '17 at 12:35
Sure, I would be happy to see other examples. @georgexsh's is pretty good one. – taras Oct 20 '17 at 12:38
@TarasMatsyk maybe one question: in your second example with calling the function multiple times with different parameters. Is it guaranteed that results are the same order as the inputs. I.e. does the first value in the solution always correspond to the input triple (1, 2, 3) ? – math Oct 20 '17 at 13:03
Yes, it should be, that is what python [zip](https://docs.python.org/2/library/functions.html#zip) function does in map_async call – taras Oct 20 '17 at 13:22

how to handover *args to pathos' amap

2 Answers2