Sharing and Updating List between Processes and Threads

Question

Trying to Share AND Update a list between Main Process - Main Thread & Sub Process - Sub Thread. Currently it appears that both the main process and main thread share the update of one copy of the list, and the sub process and sub thread share another copy of the list. I would have thought that Manager would allow the list to be shared between all 4. Any help is much appreciated.

Code:

from multiprocessing import Process, Manager
import time
import threading

def main_thread(l):
    while True:
        l[0][5] += 1
        print("main_thread:", l)
        time.sleep(6)

def sub_thread(l):
    while True:
        l[1][5] += 1
        print("sub_thread:", l)
        time.sleep(7)

def sub_process(l):
    #create a sub_thread to update l[1][5]
    trading_thread = threading.Thread(target=sub_thread, args=(l,))
    trading_thread.start()
    time.sleep(1)
    while True:
        l[0][6] += 1
        print("sub_process:", l)
        time.sleep(9)

def main_process():
    #create a thread to increment l[0][5]
    trading_thread = threading.Thread(target=main_thread, args=(l,))
    trading_thread.start()
    time.sleep(1)
    #Create a sub_process to start a sub_process and increment l[0][6]
    p = Process(target=sub_process, args=(l,))
    p.start()
    time.sleep(1)
    #I need this while loop to run as well:
    while True:
        l[0][7] += 1
        print("main_process: ", l)
        time.sleep(8)
    p.join()    #this could be a problem - we never get to this because sub process is never supposed to finish,
                #I tried to put this before the 'While' loop above but then the loop never gets triggered.

if __name__ == '__main__':
    manager = Manager()
    l = manager.list()
    l = [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 0, 0, 0],
        ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 0, 0, 0]]
    main_process()

Output:

main_thread: [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 1, 0, 0], ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 0, 0, 0]]
sub_thread: [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 1, 0, 0], ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 1, 0, 0]]
main_process:  [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 1, 1, 0], ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 0, 0, 0]]
sub_process: [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 1, 0, 0], ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 1, 1, 0]]
main_thread: [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 2, 1, 0], ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 0, 0, 0]]
sub_thread: [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 1, 0, 0], ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 2, 1, 0]]
main_process:  [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 2, 2, 0], ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 0, 0, 0]]
sub_process: [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 1, 0, 0], ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 2, 2, 0]]
main_thread: [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 3, 2, 0], ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 0, 0, 0]]
sub_thread: [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 1, 0, 0], ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 3, 2, 0]]
main_thread: [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 4, 2, 0], ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 0, 0, 0]]
main_process:  [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 4, 3, 0], ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 0, 0, 0]]
sub_process: [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 1, 0, 0], ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 3, 3, 0]]
sub_thread: [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 1, 0, 0], ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 4, 3, 0]]

Answer 1

When you do this:

    l = manager.list()
    l = [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 0, 0, 0],
        ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 0, 0, 0]]

the second assignment deletes the first assignment and replaces it with a plain old Python list. Do this:

    l = manager.list(
        [['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 0, 0, 0],
        ['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 0, 0, 0]]
    )

FOLLOWUP

The other problem is trickier. That statement does convert the first list to a managed list. If you change your threads to do l.append(7), you'll see that the list does grow. However, the INSIDE lists are still normal list objects, and they are not shared. So, l is a managed list, but l[0] is not.

You CAN work around this by doing

    l = manager.list([
        manager.list(['LACE', 1639144800000, 1.157857245, '9:00 December 10, 2021', 'True', 0, 0, 0]),
        manager.list(['ARKER', 1639109835404, 0.0, '10:00 December 10, 2021', 'True', 0, 0, 0])
    ])

But if you do that, you'll have to print them differently:

    print("sub process:", l[0], l[1])

In general, sharing data structures like this is just not a good idea. Are you aware of the implementation? Manager() creates another new process, which is a TCP server that accepts updates from the child processes and broadcasts the updates back out. It is up to 100 times slower than using a regular list.

There's no good way of globally sharing data amongst processes like this. You can try an Array in shared memory, as long as you understand the restrictions. Threads can share a global variable, but processes really need to be independent of one another. You can use a Queue to transmit results back to the mother ship, but it's not really practical to update a shared data structure like this.