Adding UIImage to CMSampleBuffer in Swift

Question

I'm searching for a way to add a custom UIImage to CMSampleBuffer which we can get in didOutput sampleBuffer in AVFoundation. I'm working on a live-streaming app and use sampleBuffer and submit the frame to the broadcaster. Then, I'm trying to add a custom UIimage on the stream now, so I'm guessing if I can add the UIimage to the samplebuffer before submitting the frame to the broadcaster, it may be possible to add an image on the live stream video. However, at this moment I'm not sure if it's possible to add the UIImage to the sample buffer, and also not sure how to add it.

Once I get CMSampleBuffer from the method, is there a way to add a UIImage to the sample buffer?

let myCustomImage = UIImage(named: "customImage")!
var newSampleBuffer: CMSampleBuffer!

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    if let myCVImageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) {
        // add a custom image to the sampleBuffer and put into the newSampleBuffer variable
    }

customImageSource?.onSampleBuffer(newSampleBuffer) // onSampleBuffer method is used for submitting the frame to the broadcaster
}

Bulat Yakupov · Answer 1 · 2022-10-21T08:54:54.893

Simple answer will be, that you can’t apply UIImage over CMSampleBuffer. You need to get CVPixelBuffer from CMSampleBuffer and CGImage from UIImage. Then you create CGContext from CVPixelBuffer and fill it using your CGImage. Now you have CVPixelBuffer with UIImage applied over it.

func draw(image: UIImage, on sampleBuffer: CMSampleBuffer) {
    guard
        let cgImage = image.cgImage,
        let frame = CMSampleBufferGetImageBuffer(sampleBuffer)
    else {
        return
    }
            
    let flags = CVPixelBufferLockFlags(rawValue: 0)
    CVPixelBufferLockBaseAddress(frame, flags)
    
    let context = CGContext(
        data: CVPixelBufferGetBaseAddress(frame),
        width: CVPixelBufferGetWidth(frame),
        height: CVPixelBufferGetHeight(frame),
        bitsPerComponent: 8,
        bytesPerRow: CVPixelBufferGetBytesPerRow(frame),
        space: CGColorSpaceCreateDeviceRGB(),
        bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.premultipliedFirst.rawValue)
            .union(.byteOrder32Little)
            .rawValue
    )
    
    let renderBounds = CGRect.zero // put your location here
    context?.draw(cgImage, in: renderBounds)
    
    CVPixelBufferUnlockBaseAddress(frame, flags)
}

Note that this method will draw your image exactly on original data of CMSampleBuffer, so no unnecessary copy, conversion or casting.

Adding UIImage to CMSampleBuffer in Swift

1 Answers1