1

I am creating a SharedMemory object, where data is being written to it in an I/O bound process for subsequent processing by a separate compute-bound process. In subsequent processing of the data I want to be able to read and write with a file-like object, so I'd like to copy to a io.BytesIO object. I'm not sure how to use the buf memory view object of the SharedMemory.

How can I copy data between a SharedMemory object and a BytesIO object?

Brian61354270
  • 8,690
  • 4
  • 21
  • 43
John
  • 10,837
  • 17
  • 78
  • 141

1 Answers1

2

Given the following example setup:

from io import BytesIO
from multiprocessing.shared_memory import SharedMemory

b = BytesIO()
s = SharedMemory(create=True, size=1024)

you can write the contents of the memory view to the bytes I/O object directly with

>>> b.write(s.buf)
1024                # N.B. bytes written equals size of shared memory

This works for any file-like object, not just BytesIO.

To do the reverse, you can extract the contents of the ByteIO object into a bytes string, and then write those bytes to a desired spot in the shared memory by indexing the memoryview object:

b.seek(0)            # change stream position to start
contents = b.read()  # read contents into bytes object

start_idx = 123      # example location
s.buf[start_idx : start_idx + len(contents)] = contents
Brian61354270
  • 8,690
  • 4
  • 21
  • 43
  • And what about the reverse process please - writing from the BytesIO into the SharedMemory memoryview? – John Mar 07 '23 at 17:22
  • 1
    @John It's an ordinary memoryview, so you can index it to read/write bytes. You can get a `bytes` object from by ByteIO seeking to the start and reading. See my edit – Brian61354270 Mar 07 '23 at 18:06