Apple has a sample code called Rosy Writer that shows how to capture video and apply effects to it.
During this section of the code, on the outputPreviewPixelBuffer part, Apple supposedly shows how they keep preview latency low by dropping stale frames.
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
CMFormatDescriptionRef formatDescription = CMSampleBufferGetFormatDescription( sampleBuffer );
if ( connection == _videoConnection )
{
if ( self.outputVideoFormatDescription == NULL ) {
// Don't render the first sample buffer.
// This gives us one frame interval (33ms at 30fps) for setupVideoPipelineWithInputFormatDescription: to complete.
// Ideally this would be done asynchronously to ensure frames don't back up on slower devices.
[self setupVideoPipelineWithInputFormatDescription:formatDescription];
}
else {
[self renderVideoSampleBuffer:sampleBuffer];
}
}
else if ( connection == _audioConnection )
{
self.outputAudioFormatDescription = formatDescription;
@synchronized( self ) {
if ( _recordingStatus == RosyWriterRecordingStatusRecording ) {
[_recorder appendAudioSampleBuffer:sampleBuffer];
}
}
}
}
- (void)renderVideoSampleBuffer:(CMSampleBufferRef)sampleBuffer
{
CVPixelBufferRef renderedPixelBuffer = NULL;
CMTime timestamp = CMSampleBufferGetPresentationTimeStamp( sampleBuffer );
[self calculateFramerateAtTimestamp:timestamp];
// We must not use the GPU while running in the background.
// setRenderingEnabled: takes the same lock so the caller can guarantee no GPU usage once the setter returns.
@synchronized( _renderer )
{
if ( _renderingEnabled ) {
CVPixelBufferRef sourcePixelBuffer = CMSampleBufferGetImageBuffer( sampleBuffer );
renderedPixelBuffer = [_renderer copyRenderedPixelBuffer:sourcePixelBuffer];
}
else {
return;
}
}
if ( renderedPixelBuffer )
{
@synchronized( self )
{
[self outputPreviewPixelBuffer:renderedPixelBuffer];
if ( _recordingStatus == RosyWriterRecordingStatusRecording ) {
[_recorder appendVideoPixelBuffer:renderedPixelBuffer withPresentationTime:timestamp];
}
}
CFRelease( renderedPixelBuffer );
}
else
{
[self videoPipelineDidRunOutOfBuffers];
}
}
// call under @synchronized( self )
- (void)outputPreviewPixelBuffer:(CVPixelBufferRef)previewPixelBuffer
{
// Keep preview latency low by dropping stale frames that have not been picked up by the delegate yet
// Note that access to currentPreviewPixelBuffer is protected by the @synchronized lock
self.currentPreviewPixelBuffer = previewPixelBuffer; // A
[self invokeDelegateCallbackAsync:^{ // B
CVPixelBufferRef currentPreviewPixelBuffer = NULL; // C
@synchronized( self ) //D
{
currentPreviewPixelBuffer = self.currentPreviewPixelBuffer; // E
if ( currentPreviewPixelBuffer ) { // F
CFRetain( currentPreviewPixelBuffer ); // G
self.currentPreviewPixelBuffer = NULL; // H
}
}
if ( currentPreviewPixelBuffer ) { // I
[_delegate capturePipeline:self previewPixelBufferReadyForDisplay:currentPreviewPixelBuffer]; // J
CFRelease( currentPreviewPixelBuffer ); /K
}
}];
}
- (void)invokeDelegateCallbackAsync:(dispatch_block_t)callbackBlock
{
dispatch_async( _delegateCallbackQueue, ^{
@autoreleasepool {
callbackBlock();
}
} );
}
After hours of trying to understand this code, my brain is smoking and I cannot see how this is done.
Can someone explain like I am 5 years old, OK, make it 3 years old, how is this code doing that?
thanks.
EDIT: I have labeled the lines of outputPreviewPixelBuffer with letters to make it easy to understand the order the code is being executed.
So, the method starts and A runs and the buffer is stored into the property self.currentPreviewPixelBuffer. B runs and the local variable currentPreviewPixelBuffer is assigned with NULL. D runs and locks self. Then E runs and changes the local variable currentPreviewPixelBuffer from NULL to the value of self.currentPreviewPixelBuffer.
This is the first thing that does not makes sense. Why would I create a variable currentPreviewPixelBuffer assign it to NULL and on the next line assign it to self.currentPreviewPixelBuffer?
The following line is even more insane. Why I am asking if currentPreviewPixelBuffer is not NULL If I just assigned it to a non NULL value on E? Then H is executed and nulls self.currentPreviewPixelBuffer?
One thing I don't get is this: invokeDelegateCallbackAsync: is asynchronous, right? if it is asynchronous then every time outputPreviewPixelBuffer method runs is to set self.currentPreviewPixelBuffer = previewPixelBuffer and dispatch a block for execution, being free to run again.
If outputPreviewPixelBuffer is fired faster, we will have a bunch of blocks piled for execution.
Due to the explanations of Kamil Kocemba, I undestand that these asynchronous blocks are testing somehow if the previous one finished executing and dropping the frames if not.
Also, what exactly is @syncronized(self) locking? Is it preventing self.currentPreviewPixelBuffer from being written or read? or is it locking the local variable currentPreviewPixelBuffer? If the block under @syncronized(self) is synchronous with relation to the scope the line at I will never be NULL because it is being set on E.
