How to use multiprocessing Value to pass image base64 string?

Question

on a project of mine I try to run Figure.savefig() on seperate multiprocessing.Process (due memory leak) but I am don't manage to allocate shared variable long enough for passing the image string.

See concept example:

import base64
from multiprocessing import Process, Value

def my_func(shared_string):
    with open("image.png", "rb") as image_file:
       
        encoded_string = base64.b64encode(image_file.read())
        shared_string.value = encoded_string.decode()

if __name__ == "__main__":
    shared_string = Value('c', b'\0'*1024*1024) # Allocate 1MB of memory for the string
    print("Before:", shared_string.value)

    p = Process(target=my_func, args=(shared_string,))
    p.start()
    p.join()

    print("After:", shared_string.value)

But I receive errors, probably because I don't understand how to set the args properly

  File "C:\Users\MyName\MyFiles\test.py", line 10, in <module>
    shared_string = Value('c', b'\0'*1024*1024) # Allocate 1MB of memory for the string
  File "C:\Users\Name\miniconda3\lib\multiprocessing\context.py", line 135, in Value
    return Value(typecode_or_type, *args, lock=lock,
  File "C:\Users\MyName\miniconda3\lib\multiprocessing\sharedctypes.py", line 74, in Value
    obj = RawValue(typecode_or_type, *args)
  File "C:\Users\MyName\miniconda3\lib\multiprocessing\sharedctypes.py", line 51, in RawValue
    obj.__init__(*args)
TypeError: one character bytes, bytearray or integer expected

I also tried to use multiprocessing.Array instead, but the image string size exceed over the limit.

Ahmed AEK · Accepted Answer · 2023-03-07T11:00:27.220

a Value is for storing a single number, you are looking for an Array instead.
for arrays you need to use the array.raw attribute instead of array.value.
only bytes arrays can be stored in a fixed size array, a unicode string cannot be stored in a bytes array because it doesn't have a fixed size (a unicode character can take up more than one byte), decode() converts between bytes and str, this is probably not what you want to do, if you really must send a unicode string you are better off using multiprocessing.Queue.
just note you only need b64encode if you are sending it over http.

the following modification to your code shows how it can be done.

 import base64
from multiprocessing import Process, Value, Array


def my_func(shared_string):
    with open("my_file", "rb") as image_file:
        encoded_string = base64.b64encode(image_file.read())
        shared_string.raw = encoded_string


if __name__ == "__main__":

    # write data to file, to make sure it is there
    with open("my_file",'wb') as f:
        orig_string = b"1"*8
        encoded_string = base64.b64decode(orig_string)
        f.write(encoded_string)

    shared_string = Array('c', b'\0' * len(orig_string))  # Allocate 8 bytes of memory for the string
    print("Before:", shared_string.raw)

    p = Process(target=my_func, args=(shared_string,))
    p.start()
    p.join()

    print("After:", shared_string.raw)

HI @AhmedAEK, your answer was really helpful after I did small change. from some reason, when you set **shared_string.raw = encoded_string** it actually been set to "shared_string.value" - so by the end I printed **shared_string.value** second question, Do I need to close/clear/release the Array variable from memory after the use? — Nitcha, Mar 08 '23 at 09:18
@Nitcha the array is reference counted, it will automatically be garbage collected when both processes can no-longer access it when it goes out of scope. — Ahmed AEK, Mar 08 '23 at 13:35

How to use multiprocessing Value to pass image base64 string?

1 Answers1