io.BytesIO is very slow. Alternatives? Optimizations?

Question

I am running a Python v3.5 script on a Raspberry Pi with a camera. The program involves recording video from the picamera and taking a sample frame from the video stream to perform operations on. Sometimes, it takes a very long time (20+ s) to deal with the byte buffer. A simplified version of the code is containing the problem area is:

import io
import picamera

camera = picamera.PiCamera()
camera.start_recording("/path/to/file.h264")
cnt = 0
while True:
    if cnt > 30:
        stream = io.BytesIO()
        camera.capture(stream, use_video_port=True, resize=(1920, 1080), format='rgba')
        cnt = 0
    else:
        cnt += 1

After a while, the time it takes to open the bytestream goes crazy. In my latest run, one instance took over 48 seconds! This figure shows a plot of the times to open the byte stream for each cycle. I performed a timing test for each line in the problematic area of the code, and I can confirm that it is the stream = io.BytesIO() line causing the delays.

When I monitor the CPU and memory of the Raspberry Pi during this task using psutils, I observe no obvious problems. CPU usage sits at 10-15%, virtual memory use ~24.2%, and 0 swap is being used.

Aside from the Python program, no other user-executed processes are running on the Pi. The hardware is running a default Raspbian installation with GUI.

Since the Python program is 1000+ lines, I am not going to include anything beyond the minimal example in this question text. If you would like to take a look at it for contextual information, please take a look at this Gist with the code.

Preliminary searches suggest that this is a known issue with BytesIO. Some old bugtracking (ca. 2014) for Python suggests that this was improved for some cases in the 3.5 release.

The questions are:

• Why is BytesIO slow here?
• Is there an alternative way to stream bytes which is faster?
• Is there a better way to use BytesIO to get what I need?

EDIT: I added a line to the loop forcing the stream to close at the end of each process using stream.close(), but this appears to have been ineffective. I still had stream opening times of 20+ seconds.

EDIT_2: I misread values in the test from the edited information and missed that values had scientific notation.

Answer 1

When BytesIO is called in a loop, it must be manually closed.

In the example, BytesIO appears to be slow due to how Python handles closing the byte stream. From the documentation for BytesIO:

A stream implementation using an in-memory bytes buffer. It inherits BufferedIOBase. The buffer is discarded when the close() method is called.

Why most users will never see this

The bytes buffer is normally not destroyed until the command is issued on exit. When the Python script is finished and the environment is deconstructed, an automatic close() is issued by iobase_exit (see line 467). It can be assumed that most users just open a byte stream in the buffer and leave it open until the script finishes. Perhaps this is not the "best" way to do it, but that is how most scripts I have seen which implement io make use of it.

When new streams are called repeatedly without closing, the buffers seem to keep piling up, occasionally requiring the system to negotiate closing them at the memory limit. The limited resources of the Raspberry Pi seem to exacerbate this. This may be measurable by doing some fancy things to plot memory use as the buffer fills up, but I don't really care about it here, and it is beyond my level of experience.

Sequential use != reentry

This should not be the case if the SAME buffer is re-entered at a later time. The IO class is protected from this edge case by issuing a runtime error. See here. This is a separate case from what I reported in the original question, since a new buffer is generated each time BytesIO is called. It is relevant to discuss this as a misinterpretation of this section of the documentation precipitated the events described in the question.

Correction of the MWE in OP

import io
import picamera

camera = picamera.PiCamera()
camera.start_recording("/path/to/file.h264")
cnt = 0
while True:
    if cnt > 30:
        stream = io.BytesIO()
        camera.capture(stream, use_video_port=True, resize=(1920, 1080), format='rgba')
        stream.close()
        cnt = 0
    else:
        cnt += 1