Python - Troubles sharing objects between processes using Manager.list

Question

I am trying to share some images between two Python processes using a Manager().list() object.
I have a main file, where I create and launch the processes:

main.py

if __name__ == "__main__":

    set_start_method("spawn")

    # creating the list
    imgs_list = Manager().list()
    killing_switch = mp.Event()

    # creating the processes
    cam_reader = mp.Process(target=cam_reader_init, args=(killing_switch, imgs_list), daemon=True)
    cam_reader.start()

    frame_analyzer = mp.Process(target=frame_analyzer_init, args=(killing_switch, imgs_list), daemon=True)
    frame_analyzer.start()

    killing_switch.wait()
    sys.exit(0)

In cam_reader, I'm reading images from a video stream, and appending them to imgs_list:

# cam_reader

def cam_reader_init(killing_switch, imgs_list):
    cam_read = CamReader(killing_switch, imgs_list)
    cam_read.run()


class CamReader:

    def __init__(self, killing_switch, imgs_list):
        self.source = 0
        self.killing_switch = killing_switch
        self.imgs_list = imgs_list

    def run(self):
        self.cap = cv2.VideoCapture(self.source, cv2.CAP_FFMPEG)

        while True:
            # read frames and send them to imgs_list
            frame = None
            ret, frame = self.cap.read()
            if frame is not None:
                self.imgs_list.append(frame)

The problem is that I'm trying to access the frames that were sent to imgs_list in the other process:

# frame_analyzer

import cv2


def frame_analyzer_init(killing_switch, imgs_list):
    frame_analyzer = FrameAnalyzer(killing_switch, imgs_list)
    frame_analyzer.run()


class FrameAnalyzer:

    def __init__(self, killing_switch, imgs_list):
        self.killing_switch = killing_switch
        self.imgs_list = imgs_list

    def run(self):

        while True:
            if len(self.imgs_list) > 0:
                frame = self.imgs_list[-1]
                key = cv2.waitKey(1)
                if key == ord('q'):
                    print("done here")
                    self.killing_switch.set()
                    break
                cv2.imshow('frames', frame)

But, while in FrameAnalyzer, it appears that self.imgs_list is always empty (the len>0 condition is never met).

What am I missing?

EDIT: Using the debugger, I noticed the list in cam_reader is getting updated (images are appended to it), while the one in frame_analyzer stays always empty. Also, I saw that the 2 instances of the list in the two processes have different values of the _id field. Could this be the source of the issue? It does seem like the processes are using two different lists somehow.

The last 2 lines apear not under you `__main__` condition, fix their indentation and try again — Ahmed AEK, Jul 18 '23 at 06:26
It was a typo in the question here (not in the code). Fixed it, thanks. — Carlo, Jul 18 '23 at 07:03

Alireza Roshanzamir · Answer 1 · 2023-07-17T22:02:34.923

0

Regardless of your OpenCV-related logic, it is important to join the subprocesses. If join is not used, the execution of the parent process will complete, resulting in the termination of the subprocesses' execution as well:

# main.py

...

if __name__ == "__main__":
    # creating the list
    imgs_list = Manager().list()

    # creating the processes
    cam_reader = mp.Process(target=cam_reader_init, args=(args, imgs_list), daemon=True)
    cam_reader.start()

    frame_analyzer = mp.Process(target=frame_analyzer_init, args=(args, imgs_list), daemon=True)
    frame_analyzer.start()

    cam_reader.join()
    frame_analyzer.join()

edited Jul 17 '23 at 22:02

answered Jul 17 '23 at 21:55

Alireza Roshanzamir

1,165
6
17

Thanks for your reply! The minimal example I posted was a little too minimal, in the real code I'm using an event to avoid the issue you pointed out. I updated the question, to avoid further confusion. – Carlo Jul 18 '23 at 06:14

Python - Troubles sharing objects between processes using Manager.list

1 Answers1