0

In my project, I use the Zoom Video SDK to receive frames in a YUV 420i format.

I'm doing an experiment trying to render these frames using the WebRTC RTCMTLNSVideoView, but I seem to have a format conversion issue, as the frames rendered on the screen appear severely mangled (see attached images).

Here are the basics of what I'm doing:

class VideoFrameZoom {
    private let data: ZMVideoSDKYUVRawDataI420
    let timestamp: CMTime
    let dimensions: CGSize

    init(_ data: ZMVideoSDKYUVRawDataI420) {
        self.data = data
        timestamp = CMTime(seconds: CACurrentMediaTime(), preferredTimescale: 1_000_000)
        dimensions = CGSize(width: Int(data.streamWidth), height: Int(data.streamHeight))
    }
}

extension VideoFrameZoom: VideoFrame {
    var pixelBuffer: CVPixelBuffer? {
        var outPixelBuffer: CVPixelBuffer?

        var planeAddresses: [UnsafeMutableRawPointer?] = [
            UnsafeMutableRawPointer(data.yBuffer),
            UnsafeMutableRawPointer(data.uBuffer),
            UnsafeMutableRawPointer(data.vBuffer),
        ]

        var planeWidths: [Int] = [
            Int(data.streamWidth),
            Int(data.streamWidth / 2),
            Int(data.streamWidth / 2),
        ]

        var planeHeights: [Int] = [
            Int(data.streamHeight),
            Int(data.streamHeight / 2),
            Int(data.streamHeight / 2),
        ]

        var planeBytesPerRow: [Int] = planeWidths

        let pixelBufferAttributes: CFDictionary? = nil
        assert(data.canAddRef())
        data.addRef()

        CVPixelBufferCreateWithPlanarBytes(
            kCFAllocatorDefault,
            Int(data.streamWidth),
            Int(data.streamHeight),
            kCVPixelFormatType_420YpCbCr8Planar,
            nil,
            0,
            3,
            &planeAddresses,
            &planeWidths,
            &planeHeights,
            &planeBytesPerRow,
            onPixelBufferReleased,
            bridge(obj: data),
            pixelBufferAttributes,
            &outPixelBuffer
        )

        return outPixelBuffer
    }
}

// ...

let buffer = videoFrameZoom.pixelBuffer!

let rtcVideoFrame = 
  RTCVideoFrame(buffer: buffer, 
                rotation: RTCVideoRotation._90, 
                timeStampNs: Int64(timeStampNs))

This frame then gets rendered by the RTCMTLNSVideoView after passing through a RTCVideoTrack

This results in frames where I can occasionally see artifacts of the original image, but clearly is not correct:

enter image description here enter image description here enter image description here

With the previous renderer we used in the project, our video renders correctly (note that the previous renderer used the same var pixelBuffer code from above), so this rules out an issue with the video source.

Should I be using a different format than kCVPixelFormatType_420YpCbCr8Planar to feed to WebRTC?

jnpdx
  • 45,847
  • 6
  • 64
  • 94

0 Answers0