How to append return value of function into list in Pool?

Question

I'm curious if it is possible to append a return value from function f(x,y) into, for example, list.

So I have this:

Parallel:

 def download_unit_LVs(self, cislo_uzemia):
     pool = Pool(number_of_workers)
     for cislo in cisla:
         pool.apply_async(self.download_cislo, args=(cislo_uzemia,cislo))
     pool.close()
     pool.join()
     self.manager.commit()

This is the way how I run the method self.download_cislo parallel but the problem is, that it returns a value which I have to append to results list.

How to do that?

Sequential:

 def download_unit_LVs(self, cislo_uzemia):
        results = []
        for cislo in cisla:
            results.append(self.download_cislo(cislo_uzemia,cislo))
        self.manager.commit()

unutbu · Accepted Answer · 2015-12-30T19:16:05.393

7

The pool.apply_async call can be passed a callback function. The callback function will be called when foo function completes and will be passed the value returned by foo. Note that the return value must be picklable, as the communication between processes is done with a Queue.

import multiprocessing as mp

def foo(x):
    return x * x

if __name__ == '__main__':
    result = []
    pool = mp.Pool()
    for i in range(100):
        pool.apply_async(foo, args=(i, ), callback=result.append)
    pool.close()
    pool.join()
    print(result)
    # [0, 1, 4, 9, 16, ... 9409, 9604, 9801]

edited Dec 30 '15 at 19:16

answered Dec 30 '15 at 12:07

unutbu

842,883
184
1,785
1,677

The `lambda` expression is unnecessary, just pass `result.append` as `callback`. In this example callbacks and the loop are also unnecessary because those can be replaced by `Pool.map()` which already returns a list with all results. – BlackJack Dec 30 '15 at 17:01
I just tested both options with `map` and `apply_async` and `apply_async` is faster, thanks! – virtualdvid Sep 25 '18 at 01:15
In your example, the list follows the correct order of the `for` loop. But when I apply it to my real case, half of the elements in the list are in the wrong order. How to make sure they append in the correct order? – Raven Cheuk Nov 06 '19 at 08:25

score 1 · Answer 2 · answered Dec 30 '15 at 17:08

If the results should end up in a list use the map() method of the Pool:

def download_unit_LVs(self, cislo_uzemia):
    pool = Pool(number_of_workers)
    results = pool.map(partial(self.download_cislo, cislo_uzemia), self.cisla)
    self.manager.commit()

partial() is from the functools module.

How to append return value of function into list in Pool?

2 Answers2