Applying a filter on captureOutput is very laggy and crashes after a specific time

Question

I have successfully implemented a filter on my camera preview. Here is the initialization, in viewDidLoad():

        sessionQueue.async {
        switch self.setupResult {
        case .success:
            // Only setup observers and start the session if setup succeeded.
            self.addObservers()

            // Buffer
            let videoOutput = AVCaptureVideoDataOutput()
            let dataOutputQueue = DispatchQueue(label: "video data queue",
                                                qos: .userInitiated,
                                                attributes: [],
                                                autoreleaseFrequency: .workItem)

            let x = DispatchQueue.main

            videoOutput.setSampleBufferDelegate(self,
                                                queue: x)

            if self.session.canAddOutput(videoOutput) {
                self.session.addOutput(videoOutput)
                self.session.startRunning()
            }

My captureOutput:

lazy var context = CIContext()

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    let videoPreviewLayerOrientation = viewPreview.videoPreviewLayer.connection?.videoOrientation
    print("capture output called")
    connection.videoOrientation = videoPreviewLayerOrientation!

    let comicEffect = CIFilter(name: "CIComicEffect")
    let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
    let cameraImage = CIImage(cvImageBuffer: pixelBuffer!)

    comicEffect!.setValue(cameraImage, forKey: kCIInputImageKey)

    let cgImage = self.context.createCGImage(comicEffect!.outputImage!, from: cameraImage.extent)!

    DispatchQueue.main.async {
        let filteredImage = UIImage(cgImage: cgImage)
        self.imageView.image = filteredImage
    }
}

Right now the applied filter works, but it is VERY laggy sometimes and I receive this messages, and after a time, the preview ends and I only receive this messages:

2020-04-08 15:59:36.242047+0200 Bumi[28519:6238030] Execution of the command buffer was aborted due to an error during execution. Caused GPU Timeout Error (IOAF code 2)

It feels like captureOutput is called A lot of times per second.

I know that I should implement the queue for videoOutput.setSampleBufferDelegate like this:

DispatchQueue(label: "video data queue",
                                    qos: .userInitiated,
                                    attributes: [],
                                    autoreleaseFrequency: .workItem)

But since I initialize it in my sessionQueue, this is not possible.

Where is my problem here?

"But since I initialize it in my sessionQueue, this is not possible" Why? You should really not use the main queue for getting _and_ processing the frames. Also, using an `UIImageView` in this scenario is not ideal – it's not meant to display this many different images per second. Try to render into an `MTKView` instead. — Frank Rupprecht, Apr 08 '20 at 15:34
Hi Frank, I will have a look at MTKView and change it. To the queue stuff: I made a video , what is happening after I open my camera on the app with the queue implemented accordingly: It is my first time implementing filters and stuff, so I do not know where to start. Maybe the video is helpful, why I can not use the queue in my session queue? Link: https://www.youtube.com/watch?v=bshOV08em9g&feature=youtu.be&fbclid=IwAR1paVtHmA6OX5fKfirJ_BExbaIE_rUHwHqDZm4VxFP2k4HdfjNfE7_kSBY — ProjektWeinheim, Apr 08 '20 at 17:42
Ah, accessing `viewPreview.videoPreviewLayer` outside of the main thread is causing the Main Thread Checker to print those warnings. The easy way around that is to keep a reference to the `videoPreviewLayer` directly (instead of through `viewPreview`) and use that instead. — Frank Rupprecht, Apr 09 '20 at 05:51

score 1 · Accepted Answer · answered Apr 09 '20 at 06:10

1

I highly recommend you check out the AVCamFilter example project from Apple. It covers efficient displaying of processed frames with MTKView and filtering video frames with either high-level Core Image or low-level Metal.

answered Apr 09 '20 at 06:10

Frank Rupprecht

9,191
31
56

Applying a filter on captureOutput is very laggy and crashes after a specific time

1 Answers1