iOS: convert UIImage to CMSampleBuffer

Question

Some questions address how to convert a CMSampleBuffer to a UIImage, but there are no answers on how to do the reverse, i.e., convert UIImage to CMSampleBuffer.

This question is different from similar ones because the code below provides a starting point for converting a UIImage to a CVPixelBuffer, which hopefully someone with more AVFoundation expertise can help fix to convert to a CMSampleBuffer.

func convertImageToBuffer(from image: UIImage) -> CVPixelBuffer? {
    let attrs = [
        String(kCVPixelBufferCGImageCompatibilityKey) : kCFBooleanTrue,
        String(kCVPixelBufferCGBitmapContextCompatibilityKey) : kCFBooleanTrue
        ] as [String : Any]
    var buffer : CVPixelBuffer?
    let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(image.size.width), Int(image.size.height), kCVPixelFormatType_32ARGB, attrs as CFDictionary, &buffer)
    guard (status == kCVReturnSuccess) else {
        return nil
    }

    CVPixelBufferLockBaseAddress(buffer!, CVPixelBufferLockFlags(rawValue: 0))
    let pixelData = CVPixelBufferGetBaseAddress(buffer!)

    let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
    let context = CGContext(data: pixelData, width: Int(image.size.width), height: Int(image.size.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(buffer!), space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)

    context?.translateBy(x: 0, y: image.size.height)
    context?.scaleBy(x: 1.0, y: -1.0)

    UIGraphicsPushContext(context!)
    image.draw(in: CGRect(x: 0, y: 0, width: image.size.width, height: image.size.height))
    UIGraphicsPopContext()
    CVPixelBufferUnlockBaseAddress(buffer!, CVPixelBufferLockFlags(rawValue: 0))

    return buffer
}

@Spads to add a text overlay to an AVAssetWriter as it's creating a video. Right now, we can create a video from the user's camera with AVAssetWriter, but as it's being created, we want to add a text overlay. Using AVVideoComposition after the video gets created is too slow. Any clues? — Crashalot, Dec 20 '17 at 07:43
@Spads nope, didn't see it! but it seems like there are issues with this approach? thanks for finding it, though. appreciate your help! any other ideas? will also separately dig more into AVAssetWriterInputPixelBufferAdaptor. — Crashalot, Dec 20 '17 at 09:07
@Spads no problem, this is better than nothing. thanks again. :) — Crashalot, Dec 20 '17 at 10:07

score 3 · Answer 1 · answered Jan 26 '22 at 06:59

You are halfway done. Just convert CVPixelBuffer to CMSampleBuffer:

extension UIImage {
    
    var cvPixelBuffer: CVPixelBuffer? {
        let attrs = [
            String(kCVPixelBufferCGImageCompatibilityKey): kCFBooleanTrue,
            String(kCVPixelBufferCGBitmapContextCompatibilityKey): kCFBooleanTrue
            ] as [String: Any]
        var buffer: CVPixelBuffer?
        let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(self.size.width), Int(self.size.height), kCVPixelFormatType_32ARGB, attrs as CFDictionary, &buffer)
        guard status == kCVReturnSuccess else {
            return nil
        }

        CVPixelBufferLockBaseAddress(buffer!, CVPixelBufferLockFlags(rawValue: 0))
        let pixelData = CVPixelBufferGetBaseAddress(buffer!)

        let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
        let context = CGContext(data: pixelData, width: Int(self.size.width), height: Int(self.size.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(buffer!), space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)

        context?.translateBy(x: 0, y: self.size.height)
        context?.scaleBy(x: 1.0, y: -1.0)

        UIGraphicsPushContext(context!)
        self.draw(in: CGRect(x: 0, y: 0, width: self.size.width, height: self.size.height))
        UIGraphicsPopContext()
        CVPixelBufferUnlockBaseAddress(buffer!, CVPixelBufferLockFlags(rawValue: 0))

        return buffer
    }
       
    func createCMSampleBuffer() -> CMSampleBuffer? {
        let pixelBuffer = cvPixelBuffer
        var newSampleBuffer: CMSampleBuffer?
        var timimgInfo: CMSampleTimingInfo?
        var videoInfo: CMVideoFormatDescription?
        CMVideoFormatDescriptionCreateForImageBuffer(allocator: nil, imageBuffer: pixelBuffer!, formatDescriptionOut: &videoInfo)
        CMSampleBufferCreateForImageBuffer(allocator: kCFAllocatorDefault,
                                           imageBuffer: pixelBuffer!,
                                           dataReady: true,
                                           makeDataReadyCallback: nil,
                                           refcon: nil,
                                           formatDescription: videoInfo!,
                                           sampleTiming: &timimgInfo,
                                           sampleBufferOut: &newSampleBuffer)
        return newSampleBuffer!
    }
    
}

I'm getting an error " Unexpectedly found nil while unwrapping an Optional value" for "timimgInfo" in "sampleTiming". how can I fix this — Vishnu Hari, Dec 29 '22 at 10:10

iOS: convert UIImage to CMSampleBuffer

1 Answers1