CVPixelBuffer – How to capture every THIRD frame out of 60 fps?

Question

I need only 20 frames out of 60 frames per second for processing (CVPixelBuffer).

How to capture every third ARFrame in ARKit session? I need approximately 20 fps for capture (I understand there may be a drop frame).

Here's a code's snippet:

func updateCoreML() {

    let pixelBuffer: CVPixelBuffer? = (sceneView.session.currentFrame?.capturedImage)

    if pixelBuffer == nil { return }
    let ciImage = CIImage(cvPixelBuffer: pixelBuffer!)
    let imageRequestHandler = VNImageRequestHandler(ciImage: ciImage, options: [:])

    do {
        try imageRequestHandler.perform(self.visionRequests)
    } catch {
        print(error)
    }
}

Maxim Volgin · Accepted Answer · 2018-12-03T08:12:06.593

The simplest way known to me is to use RxARKit and apply .throttle() operator to session.rx.didUpdateFrame. I believe it is not prudent to skip every so many events because frame rate is not guaranteed to be 60fps at all times, so it is best to use .throttle() to get a frame at most every so many milliseconds, no matter what the actual framerate is. You can plug the result of it into RxVision, which will take care that that very frame will be used by CoreML.

import RxSwift
import RxARKit
import RxVision


let disposeBag = DisposeBag()
let mlRequest: RxVNCoreMLRequest<CVPixelBuffer> = VNCoreMLRequest.rx.request(model: model, imageCropAndScaleOption: .scaleFit)

mlRequest
    .observable
    .subscribe { [unowned self] (event) in
        switch event {
            case .next(let completion):       
                let cvPixelBuffer = completion.value
                if let result = completion.request.results?[0] as? VNClassificationObservation {
                    os_log("results: %@", type: .debug, result.identifier)
                }
            default:
                break
        }
    }
    .disposed(by: disposeBag)

session
    .rx
    .didUpdateFrame
    .throttle(1/20)
    .subscribe { event in
        switch event {
        case .next(let didUpdateFrame):
        let frame: ARFrame = didUpdateFrame.frame        
        let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: frame.capturedImage, orientation: .up, options: requestOptions)
        do {
            try imageRequestHandler.rx.perform([mlRequest], with: frame.capturedImage) } catch {
            print(error)
        }            
        break
        default:
            break
        }
    }
    .disposed(by: disposeBag)

I am using RxVision in my app that is published on AppStore, probably RxARKit is no problem either. — Maxim Volgin, Dec 03 '18 at 07:42

score 0 · Answer 2 · answered Feb 13 '19 at 13:36

You can either set custom frame rate, so you will get only 20 frames, or you can receive all frames but use only every third frame by taking a flag variable and incrementing it.

  func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
       totalFrameCount += 1
       if totalFrameCount % 3 != 0{ return }
}

or you can set custom frame rate by extending AVCaptureDevice

 lockForConfiguration()
 activeVideoMinFrameDuration = CMTimeMake(1, 20)
 activeVideoMaxFrameDuration = CMTimeMake(1, 20)
 unlockForConfiguration()

Tahir, thanks for you answer. But this question is exceptionally about ARKit, not AVFoundation. I know what AVFoundation can do and what way. — Andy Jazz, Feb 13 '19 at 15:00

CVPixelBuffer – How to capture every THIRD frame out of 60 fps?

2 Answers2