Sharing a custom class object through multiprocessing manager

Question

I am working on a project where I have very specific kind of logs and for this logs to work I have created a class which basically maintains a dictionary that logs time and actions taken for each step of the execution. Which is later used for multiple analytical purposes. later I create a JSON file from this dictionary and upload it to s3 bucket.

Everything worked fine when I had only one main process. But for the efficiency, I have included multiprocessing for a specific task. I am generating 4 processes which call the same function and I want the object of my class which maintains the logs to be shared by all of these processes.

I have gone through the other StackOverflow questions but none of them seemed to work. Please guide me if you think this has a better answer somewhere and I might have skipped it.

class JsonLogs:
    def __init__(self, date, site_name, user_uuid, is_login=False, request_data=None):
        self.date = date
        self.attribute = value
        if is_login:
            ...
        else:
            ...


    def add_a_logs(self, process, message):
        self.log_dict['key']['some_key'].append(
            {
             'start_time': datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f'),
             'start_message': message,
             'end_message': None,
             'end_time': None,
             'duration': None

            }
        )

    def update_a_logs(self, process=None, message=None, response_data=None):
        log_obj = next((log for log in self.log_dict["key"]['some_key'] if log['process'] == process),
                       None)

        if response_data:
            self.log_dict['key']['some_key'] = response_data

        if log_obj:
            log_obj['end_time'] = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')
            log_obj['end_message'] = message
            td = datetime.datetime.now() - datetime.datetime.strptime(log_obj['start_time'], '%Y-%m-%d %H:%M:%S.%f')

            log_obj['duration'] = "{}".format(td.seconds)

    .....
    .....

There are also other methods like this and then this dictionary is converted to JSON and uploaded in s3 bucket. This has nothing to do with the python logging module as I have a very specific requirement.

I need the object of this class to be shared by 4 processes that I am creating.

EDIT

To make things easier for someone to understand, what I am really trying to achieve can be understood with the two examples given below.

Single process: This works

from multiprocessing import Process


class A:
    def __init__(self, a):
        self.a = a

def a():
    x = A(10)
    b(x)
    print(x.a)

def b(y):
    y.a = 20

a()

OUTPUT: 20

Multi-Processing: What I am trying to achieve

from multiprocessing import Process


class A:
    def __init__(self, a):
        self.a = a


def a():
    x = A(10)
    p1 = Process(target=b, args=(x,))
    p2 = Process(target=b, args=(x,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    # b(x)
    print(x.a)

def b(y):
    y.a = 20


a()

Current-OUTPUT: 10

What I Need: 20

Answer 1

You need to share objects between processes using shared memory since each process has its own memory space. You can share objects using a manager object. But it is better to avoid sharing complex objects as it is stated by the guideline.

Alternatively you can use a queue object. Here is a toy example:

import datetime
from multiprocessing import Process, Queue

def worker(q):
    for _ in range(5):
        q.put(
        {
            'start_time': datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f'),
            'start_message': 'message'
        })
    q.put(None) # End of the queue

def a():
    q = Queue()
    p1 = Process(target=worker, args=(q,))
    p2 = Process(target=worker, args=(q,))
    p1.start()
    p2.start()

    worker_end = 0
    while True:
        i = q.get()
        if i is None:
            worker_end += 1
        else:
            print(i)

        if worker_end == 2:
            break

    p1.join()
    p2.join()

a()