Converting CVPixelBuffer to Mat (OpenCV)

Question

UPDATE seems issue with LegoCV, it can't even create OCVMat from simple UIImage

    let image = UIImage(named: "myImage")
    let mat = OCVMat(image: image!)

I'm trying to convert CVPixelBuffer (from a Camera - Video Output) to a Mat (OpenCV) - OCVMat

I use the next framework to add OpenCV in my iOS Swift project https://github.com/Legoless/LegoCV

it wraps OpenCV native C++ classes into lightweight Objective-C classes, which are then natively bridged to Swift

I implemented AVCaptureVideoDataOutputSampleBufferDelegate for my camera class to get camera frame buffers

set needed video input and started a session, camera works ok, buffers are coming, but when I'm trying to create OCVMat from CVPixelBuffer ("OCVMat(pixelBuffer: imageBuffer)") app crashes with next error

opencv(1934,0x16f087000) malloc: *** error for object 0x16f086108: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

Objective-C class: https://github.com/Legoless/LegoCV/blob/master/LegoCV/LegoCV/Wrapper/Core/Mat/OCVMatDataAllocator.mm

Some Swift Code

fileprivate func configureVideoOutput() {
    let videoOutput = AVCaptureVideoDataOutput()
    videoOutput.setSampleBufferDelegate(self, queue: DispatchQueue(label: "sample buffer"))

    if self.session.canAddOutput(videoOutput) {
        print("canAddOutput yes")
        self.session.addOutput(videoOutput)
        print("canAddOutput yes added")
    } else {
        print("canAddOutput no")
    }
}

private func matFromSampleBuffer(sampleBuffer: CMSampleBuffer) -> OCVMat? {
    guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return nil }
    return OCVMat(pixelBuffer: imageBuffer)
}

public func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    count = count + 1
    print("Got a frame # \(count)")
    guard let mat = matFromSampleBuffer(sampleBuffer: sampleBuffer) else { return }
}

We are considering this framework for some standard image manipulation needed in our iOS app. Is it recommended? Did you find a solution? — Arik Segal, Jan 31 '18 at 12:54
@ArikSegal still not, I'm going to try https://github.com/oaleeapp/SwiftEyes mb it will work — user924, Jan 31 '18 at 18:03
@ArikSegal actually that one is just a plagiarism of LegoCV and it also doesn't work — user924, Feb 02 '18 at 11:16

user924 · Accepted Answer · 2018-07-15T18:51:47.230

1

solution - don't use wrappers, it's better to write OpenCV C++ code in .mm file (mix it with Objective-C)

edited Jul 15 '18 at 18:51

answered Apr 10 '18 at 18:16

user924

8,146
7
57
139

If you're using OpenCV/C++ code to convert between CVPixelBuffer to Mat, I've found that the accepted answer on this SO post to be the solution that worked for me: https://stackoverflow.com/questions/34677890/convert-cmsamplebufferref-to-cvmat – Drew H Jan 16 '23 at 18:11

Converting CVPixelBuffer to Mat (OpenCV)

1 Answers1