Python Multiprocessing: How to make Pool and Manager work together?

Question

I am trying to understand how Pool() and Manager() can be combined in Python's multiprocessing library for having shared objects between the individual worker processes. In my example, I would like to have a shared list input_list, which should be accessable for all worker processes:

from multiprocessing import Pool, Manager

def f(x):
    input_list.append(x)
    return x**2

if __name__ == '__main__':
    manager = Manager()
    input_list = manager.list()
    with Pool(processes=3) as pool:
        results = pool.map(f, range(10))
    print(input_list)

However, I get a NameError: name 'input_list' is not defined. Any idea why input_list is not recognized as a shared object?

your example works fine when I run on python 3.8. it returns [0, 2, 3, 4, 5, 6, 7, 8, 9, 1]. — tomanizer, Jul 31 '20 at 10:12
Interesting. I tried with python 3.8.2, still get the same error. Are you also working on a Windows machine? — proedig, Jul 31 '20 at 12:08

proedig · Answer 1 · 2020-07-31T12:02:02.783

I found a solution using functools. The shared list needs to be passed to the worker function as an additional argument:

from multiprocessing import Pool, Manager
import functools

def f(x, input_list):
    input_list.append(x)
    return x**2

if __name__ == '__main__':
    manager = Manager()
    input_list = manager.list()
    with Pool(processes=3) as pool:
        results = pool.map(functools.partial(f, input_list=input_list), range(10))
    print(input_list)

An exemplary output is then:

[0, 3, 1, 4, 2, 5, 6, 7, 8, 9]

Python Multiprocessing: How to make Pool and Manager work together?

1 Answers1