6

When using AVCaptureVideoDataOutput and defining a sample buffer delegate with a dispatch queue (setSampleBufferDelegate:queue), we are experiencing on iOS 8 that AVFoundation does not post the sample buffers on the specified dispatch queue but rather always uses "com.apple.avfoundation.videodataoutput.bufferqueue".

This works as expected on iOS7.

Has anyone else experienced this?

An obvious workaround is to manually call dispatch_sync in the callback to sync processing to the custom dispatch queue, but this, strangely, causes a deadlock...

Sample code that produces this issue:

- (void)viewDidLoad {
    [super viewDidLoad];
    // Do any additional setup after loading the view, typically from a nib.

    AVCaptureSession *session = [[AVCaptureSession alloc] init];
    session.sessionPreset = AVCaptureSessionPresetMedium;

    AVCaptureVideoPreviewLayer *captureVideoPreviewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:session];
    captureVideoPreviewLayer.frame = self.view.bounds;
    [self.view.layer addSublayer:captureVideoPreviewLayer];

    [session addInput:[AVCaptureDeviceInput deviceInputWithDevice:[AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo] error:nil]];

    AVCaptureVideoDataOutput *output = [[AVCaptureVideoDataOutput alloc] init];

    queue = dispatch_queue_create("our.dispatch.queue", DISPATCH_QUEUE_SERIAL);

    [output setSampleBufferDelegate:self queue:queue];

    [session addOutput:output];

    [session startRunning];
}

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
    NSLog(@"Running on queue %@, queue that was set is %@, this is %s", dispatch_get_current_queue(),
      [captureOutput performSelector:@selector(sampleBufferCallbackQueue)],
      queue == dispatch_get_current_queue() ? "our queue" : "not our queue!!!");
}
m1h4
  • 1,139
  • 2
  • 13
  • 22

2 Answers2

7

What's probably happening here is that their queue, com.apple.avfoundation.videodataoutput.bufferqueue, has been set to target yours using dispatch_set_target_queue. This is functionally equivalent to dispatching to your queue, but would explain the name, and would also explain the deadlock when you tried to dispatch back to your queue.

In other words, just because the queue name isn't equal to your queue's name doesn't mean the block isn't executing on your queue.

ipmcc
  • 29,581
  • 5
  • 84
  • 147
  • yes, this also crossed our minds, _but_ the whole issue was initially discovered when we had a problem/race-condition with multiple threads accessing a shared resource which should only be accessed from our processing queue. the stack trace showed one thread running a job on our processing queue and another running the "com.apple.avfoundation.videodataoutput.bufferqueue". if the avfoundation queue was dispatch_sync-ing to our queue this should not have happened since our queue is serial – m1h4 Oct 02 '14 at 16:39
  • 2
    You could be sure by suspending your queue beforehand. Then if you end up in the block, you can be sure it's a bug. – ipmcc Oct 02 '14 at 18:29
  • 1
    Suspending "our" queue actually stops the processing queue so it seems you were correct, "com.apple.avfoundation.videodataoutput.bufferqueue" is using our queue as a target queue. Still, the reason why we had two simultanious threads running operations that should have been queued in the same serial queue (one was noted as from "com.apple.avfoundation.videodataoutput.bufferqueue" and the other from "our.dispatch.queue") is a mistery to us. This should not have happened, otherwise the only explanation is that there's some sort of synchronization oversight in avfoundation's queue handling. – m1h4 Oct 03 '14 at 14:46
  • You can also use `dispatch_queue_set_specific` and `dispatch_queue_get_specific` to determine targeting relationships. Set a specific value on your serial queue and then query it from both of the call sites. If you get back your tag value from both call sites and they're being called concurrently, then something squirrelly is going on. – ipmcc Oct 03 '14 at 14:52
  • Thanks for the suggestion and helping out! Since this does not occur that often, we will place a NSLog with the current dispatch queue label and our "specific" set on the queue at the shared resource location so we can see who exactly called it (it should only be called by code from "our" queue). I will mark your answer as accepted since it does answer my original question. – m1h4 Oct 03 '14 at 15:00
  • Any updates on this? My workaround is, in `-captureOutput:` to `CFRetain(sampleBuffer);` and then `dispatch_async` to the queue that I passed to `-setSampleBufferDelegate:`. In the block I pass to dispatch_async, I have to `CFRelease(sampleBuffer);` when I am done with it. – mahboudz Aug 24 '15 at 20:15
0

To get around this issue, I had to modify my -captureOutput::

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    dispatch_queue_t queue = ((MyAppDelegate *)UIApplication.sharedApplication.delegate).videoDataOutputQueue;
    CFRetain(sampleBuffer);
    dispatch_async(queue, ^{

        for (id<AVCaptureVideoDataOutputSampleBufferDelegate> target in captureTargets.copy)
            [target captureOutput:captureOutput didOutputSampleBuffer:sampleBuffer fromConnection:connection];
        CFRelease(sampleBuffer);
    });
}
mahboudz
  • 39,196
  • 16
  • 97
  • 124
  • what is captureTarget.copy here.? – Mubin Mall Jan 03 '18 at 21:02
  • 1
    I have a number of different users of the images coming from the camera. Those "users" are stored in the array captureTargets. Since the array can change in another queue, I use a copy of the array to loop through and send the camera output to each. – mahboudz Jan 19 '18 at 07:53