In my project, I use the Zoom Video SDK to receive frames in a YUV 420i format.
I'm doing an experiment trying to render these frames using the WebRTC RTCMTLNSVideoView
, but I seem to have a format conversion issue, as the frames rendered on the screen appear severely mangled (see attached images).
Here are the basics of what I'm doing:
class VideoFrameZoom {
private let data: ZMVideoSDKYUVRawDataI420
let timestamp: CMTime
let dimensions: CGSize
init(_ data: ZMVideoSDKYUVRawDataI420) {
self.data = data
timestamp = CMTime(seconds: CACurrentMediaTime(), preferredTimescale: 1_000_000)
dimensions = CGSize(width: Int(data.streamWidth), height: Int(data.streamHeight))
}
}
extension VideoFrameZoom: VideoFrame {
var pixelBuffer: CVPixelBuffer? {
var outPixelBuffer: CVPixelBuffer?
var planeAddresses: [UnsafeMutableRawPointer?] = [
UnsafeMutableRawPointer(data.yBuffer),
UnsafeMutableRawPointer(data.uBuffer),
UnsafeMutableRawPointer(data.vBuffer),
]
var planeWidths: [Int] = [
Int(data.streamWidth),
Int(data.streamWidth / 2),
Int(data.streamWidth / 2),
]
var planeHeights: [Int] = [
Int(data.streamHeight),
Int(data.streamHeight / 2),
Int(data.streamHeight / 2),
]
var planeBytesPerRow: [Int] = planeWidths
let pixelBufferAttributes: CFDictionary? = nil
assert(data.canAddRef())
data.addRef()
CVPixelBufferCreateWithPlanarBytes(
kCFAllocatorDefault,
Int(data.streamWidth),
Int(data.streamHeight),
kCVPixelFormatType_420YpCbCr8Planar,
nil,
0,
3,
&planeAddresses,
&planeWidths,
&planeHeights,
&planeBytesPerRow,
onPixelBufferReleased,
bridge(obj: data),
pixelBufferAttributes,
&outPixelBuffer
)
return outPixelBuffer
}
}
// ...
let buffer = videoFrameZoom.pixelBuffer!
let rtcVideoFrame =
RTCVideoFrame(buffer: buffer,
rotation: RTCVideoRotation._90,
timeStampNs: Int64(timeStampNs))
This frame then gets rendered by the RTCMTLNSVideoView
after passing through a RTCVideoTrack
This results in frames where I can occasionally see artifacts of the original image, but clearly is not correct:
With the previous renderer we used in the project, our video renders correctly (note that the previous renderer used the same var pixelBuffer
code from above), so this rules out an issue with the video source.
Should I be using a different format than kCVPixelFormatType_420YpCbCr8Planar
to feed to WebRTC?