2

We are using ReplayKit to get CMSSampleBuffers and forward them to OTVideoCaptureConsumer and after resizing ReplayKit buffers into a 1280x720 or 1024x768 buffer, we get a screen share feed, but it always becomes blurry.

After making sure we controlled the FPS sent to the videoCaptureConsumer to limit the bandwidth, we are stil unable to get a good quality image received from the media router on subscribed devices.

We validated that the image sent to OpenTok OTVideoCaptureConsumer is a clear image but the resulting stream always becomes blurry very quickly on the subscriber's end

Question: Anyone has figured out a way to get a clean OTPublisher screen sharing feed using the OpenTok iOS SDK regardless of the device's screen size?

Attempts:

  • Different way to send frames:

I tried sending images using CVPixelBuffer and OTVideoFrame. The sample suggested by OpenTok support uses both methods CVPixelBuffer for the camera images and OTVideoFrame for the black images when no frames are received from the camera. But the resulting stream becomes blurry either way

  • Different format and sizes:

I did not find any documentation mentioning the specs of the image expected by the iOS SDK. Is there anything available somewhere to help us understand what are the supported image format, data size limitation and supported resolutions?

I tried sending the ReplayKit buffer as is to the videoCaptureConsumer from our app and on e.g.: a 9.7 Pro iPad the native screen resolution is 1920x1440, which seems accepted by the iOS SDK (returns success), but does only render a black screen on the subscriber end.

I just don't have any other idea

  • Retransmitting the last frame when ReplayKit stops sending frames:

I saw this behaviour in the Twilio ReplayKit sample, but should we resend the last frame when ReplayKit does not send any frame? I did not seen any doc if it's ok or if the media router will behave properly if we do so

PS: Some references I found in case it might help someone else

All I've seen in the OpenTok's support suggested sample are:

  • the supported presets for the iOS AVCaptureSession which are then forwarded to the OTVideoCapturer (Custom-Video-Driver/Lets-Build-OTPublisher/Single-Cam-Capturer/TBExampleVideoCapture.m#L255)

  • the hardcoded size for the black frames (Custom-Video-Driver/Lets-Build-OTPublisher/Single-Cam-Capturer/TBExampleVideoCapture.m#L552)

  • the non processed CMSampleBuffer forwarded directly to the videoCaptureConsumer (Custom-Video-Driver/Lets-Build-OTPublisher/Single-Cam-Capturer/TBExampleVideoCapture.m#L711)

From OpenTok's doc and forum posts:

I also found some other posts that helped understand what might be the problem:

Christian Nadeau
  • 61
  • 4
  • Do you have any updates on that? I am having the same problem with the latest update and the broadcast extension is passing the memory limit all the time. – Mehmet Baykar Aug 31 '21 at 15:31

1 Answers1

0

I ended up using the OTVideoFrame to send the image and implementing a resend last frame logic using a DispatchWorkItem

    // We just received a frame, cancel any retransmit
    self.resetRetransmitTimer()

    let resized = cgImage
    self.checkPixelBufferSize(forImage: resized)
    let ref = self.fillPixelBuffer(withCGImage: resized)
    CVPixelBufferLockBaseAddress(ref, CVPixelBufferLockFlags.readOnly)
    
    self.videoFrame?.format?.pixelFormat = .ARGB
    self.videoFrame?.timestamp = timestamp
    self.videoFrame?.format?.estimatedCaptureDelay = 10
    self.videoFrame?.format?.estimatedFramesPerSecond = self.config.desiredFrameRate
    self.videoFrame?.orientation = .up
    
    self.videoFrame?.clearPlanes()
    self.videoFrame?.planes?.addPointer(CVPixelBufferGetBaseAddress(ref))
    
    self.videoCaptureConsumer?.consumeFrame(self.videoFrame!)
    
    self.lastReplayKitImage = cgImage
    self.lastReplayKitImageTimeStamp = timestamp
    
    CVPixelBufferUnlockBaseAddress(ref, CVPixelBufferLockFlags.readOnly)
    
    self.scheduleRetransmitTimer()
Christian Nadeau
  • 61
  • 4