Sharing a list between processes with multiprocessing.Manager

Question

1st ''' import multiprocessing as mp

def func( list):
    x = 5
    list = list*5
    print('list_in_sub: ', list)

if __name__ == "__main__":
    list = mp.Manager().list(range(2))  
    p1 = mp.Process(target=func, args=( list,))
    p1.start()
    p1.join()

    print('list_in_main: ', list)

''' result: list_in_sub: [0, 1, 0, 1, 0, 1, 0, 1, 0, 1] list_in_main: [0, 1]

2nd ''' import multiprocessing as mp

def func(list):
    x = 5
    list = list.append(x)
    print('list_in_sub: ', list)

if __name__ == "__main__":
    list = mp.Manager().list(range(2))  
    p1 = mp.Process(target=func, args=(list,))
    p1.start()
    p1.join()

    print('list_in_main: ', list)

''' result: list_in_sub: None list_in_main: [0, 1, 5]

Every process runs in its own private memory-space, so global variables cannot be shared between them. The purpose of `multiprocessing.Manager`s is to lets you share certain types of data-structures between them if you pass them to the other process as arguments. I can't tell for sure what you're doing in your code, but it looks like maybe your are assuming the something declared `global` can be accessed from the other process — which will not work for the reason initially stated. — martineau, Sep 02 '21 at 08:12
Mr martineau, thank you very much. but i'm not assuming the global variable can be accessed from the other process . i've edit my code to make it easier to understand. i use the global because the callback function is packaged by the third party that i cannot change it . and this callback funciton cannot allowed to transfer variables. so i have to use global variable to make sure the variable was tranfered in each process alone. — xibuzuoluo, Sep 02 '21 at 13:38
This code is more complex than needed. I suggest you simplify the example so that it highlights the fundamental issues, i.e. sharing a list across processes. Also, the code cannot be run because it contains several undefined functions. Please make it reproducible and clearly state what you would expect to happen. — Kota Mori, Sep 02 '21 at 13:53
Well,my comment about processes having separate memory-spaces would apply to the callback functions, too — they would also need to be compatible with multiprocessing in the sense that the callback functions shouldn't require any information for another process. I agree with @KotaMori, it would be best if you supplied a runnable [mre] (MRE). — martineau, Sep 02 '21 at 15:14
ok thank you both, i'd like to try my best to give a MRE later. — xibuzuoluo, Sep 02 '21 at 22:06
@KotaMori and martineau, i've got a MRE one, but i still cannot share the variables normally, especially the 1st one, i find if i calcaluted about the shared variables in the sub processing, i cannot share the changed one in the main process. — xibuzuoluo, Sep 03 '21 at 03:31

Kota Mori · Answer 1 · 2021-09-03T08:49:26.053

The first one should be:

import multiprocessing as mp

def func(list):
    x = 5
    list *= 5
    print('list_in_sub: ', list)

if __name__ == "__main__":
    list = mp.Manager().list(range(2))  
    p1 = mp.Process(target=func, args=( list,))
    p1.start()
    p1.join()

    print('list_in_main: ', list)

# list_in_sub:  [0, 1, 0, 1, 0, 1, 0, 1, 0, 1]
# list_in_main:  [0, 1, 0, 1, 0, 1, 0, 1, 0, 1]

The second one should be:

import multiprocessing as mp

def func(list):
    x = 5
    list.append(x)
    print('list_in_sub: ', list)

if __name__ == "__main__":
    list = mp.Manager().list(range(2))  
    p1 = mp.Process(target=func, args=(list,))
    p1.start()
    p1.join()

    print('list_in_main: ', list)

# list_in_sub:  [0, 1, 5]
# list_in_main:  [0, 1, 5]

Explanation:

For the first one, list = list*5 defines a new variable list inside the function. Since this is different from the list received, the contents are different. In contrast, list *= 5 updates the variable in place.
The second one is a bug. .append method adds a new value to the list in place, and returns nothing. Hence, list = list.append(x) adds x to the original list, and defines a new variable list as None inside the function.
Note: list is a reserved keyword; you may want to use different variable names to avoid errors.

The example above may not be that interesting because the list is updated only by one process. The following demonstrates that two processes share and update the same list in a random order.

import multiprocessing as mp

def add_positive(lst):
    for i in range(1, 31):
       lst.append(i)

def add_negative(lst):
    for i in range(1, 31):
       lst.append(-i)
        
if __name__ == "__main__":
    lst = mp.Manager().list([0])  
    p1 = mp.Process(target=add_positive, args=(lst,))
    p2 = mp.Process(target=add_negative, args=(lst,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

    print('Result: ', lst)

#Result (order should vary):  
#[0, 1, 2, 3, 4, -1, 5, -2, 6, -3, 7, -4, -5, 8, -6, 9, -7, 10, -8, 
#11, -9, 12, -10, 13, -11, -12, 14, -13, 15, -14, 16, -15, 17, -16, 
#18, -17, 19, -18, 20, -19, 21, -20, -21, 22, -22, 23, -23, 24, -24, 
#25, -25, 26, -26, 27, -27, 28, -28, 29, -29, 30, -30]

Sharing a list between processes with multiprocessing.Manager

1 Answers1