2

I have an app where I use to keep frames (CVPixelBuffer) coming from the iOS camera. I use the delegate method -(void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer (CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection .

Since recent updates of iOS no more frames arrive through the delegate method if the previous once are not released. Now CVPixelBufferUnlockBaseAddress and CVPixelBufferRelease need to be called on the stored frames before new ones come.

Any ideas of why's that? I would think there was some hardware optimization but I could not find details.

focs
  • 182
  • 2
  • 10

1 Answers1

3

The documentation for this delegate method details exactly this situation:

To maintain optimal performance, some sample buffers directly reference pools of memory that may need to be reused by the device system and other capture inputs. This is frequently the case for uncompressed device native capture where memory blocks are copied as little as possible. If multiple sample buffers reference such pools of memory for too long, inputs will no longer be able to copy new samples into memory and those samples will be dropped.

If your application is causing samples to be dropped by retaining the provided CMSampleBufferRef objects for too long, but it needs access to the sample data for a long period of time, consider copying the data into a new buffer and then releasing the sample buffer (if it was previously retained) so that the memory it references can be reused.

This should not be surprising. The video capture system doesn't want to spend time allocating new memory for every frame. Memory allocation, particularly of large blocks, is very expensive.

Community
  • 1
  • 1
Rob Napier
  • 286,113
  • 34
  • 456
  • 610
  • 1
    Having worked with quite a lot with the camera feed, I can add a bit to this. On the iPhone 7 at least, the camera has a buffer pool of around 13 frames. If you manage to keep your processing within 25ms, you can safely use the buffers from the camera. If not, you either implement a producer/consumer mechanism that allows you drop frames (therefore returning them to the camera system without processing), or you make a copy of each frame and return the original. Either way, you need to release the camera frames... otherwise you have a memory leak. – CatalinM Sep 20 '17 at 14:59