2

I was looking into replaykit here https://developer.apple.com/videos/play/wwdc2018/601/ and was interested in knowing whether it would be possible to stream the result of a Metal renderpass to youtube. The use case involves giving the user additional visuals that an audience should not be able to see. So I would start by rendering only the shared parts of my graphical scene first, stream THAT to youtube, and then finish rendering the user-only content. I think this would require VideoToolbox as well if replaykit does not accept custom video, which would be disappointing. I am trying to avoid writing my own network code.

After rendering, somehow I'd need to convert the first texture to a CVPixelBuffer, I think. Sort of the way it's described here though that's not quite right for my use case.

That's where I get kind of lost. I think I probably need to copy the pixel buffer as well since that probably gets overwritten after Metal renders a frame.

Insert some use of BlitEncoder somewhere.

May I have help getting started? The documentation is rather sparse, and I think it's unclear how to combine all these steps.

synchronizer
  • 1,955
  • 1
  • 14
  • 37

1 Answers1

1

Looking at the API, I don't think it's possible to stream arbitrary content, at least not directly.

The API forces you to record the whole screen (while prompting user if he actually wants to do that). But you can post-process it afterwards using the handler you pass to the startCapture method. In that handler, however, you have accesss to CMSampleBuffer, and if it's a RPSampleBufferType.video, then you can grab your MTLTexture for the right timestamp and basically draw into it.

Here's a part where it gets really confusing. CVImageBuffer, CVPixelBuffer are just typealiases for CVBuffer.

To draw into CMSampleBuffer, you can grab CVImageBuffer from the CMSampleBuffer using its property imageBuffer, then CVPixelBufferLockBaseAddress it and then you have a linear bitmap, basically, in memory. From there, you just need to figure out a way to blit out the MTLTexture into this linear memory using the right formats, unlock the base address for the CVPixelBuffer and proceed to next one.

JustSomeGuy
  • 3,677
  • 1
  • 23
  • 31
  • Thanks. That’s certainly complicated — it sounds like I might need multiple Metal textures for multiple corresponding buffer frames at a time, or...? – synchronizer Jun 16 '21 at 12:48
  • Yep, I think so. As I understand, `CMSampleBuffer`s can sort of be "out of order" - meaning they will get reassembled into actual video stream later using the timestamps you provided. You may want to have a pool of textures, basically like video decoding works, where you have a surplus of "frames" you can show, even though it's not the time for them yet. Wish I could be more concrete, but I haven't worked with that specific API. – JustSomeGuy Jun 16 '21 at 16:06
  • Oof. I don't think regular youtube members can stream without Replaykit, so I wonder whether there's a non-Replaykit method for other services. It's challenging to do this all without sample code. My understanding is that it involves using videotoolbox and even then I'd have to figure how to compress the textures into streamable video. I find it strange that there's no automatic way to do this via some api. – synchronizer Jun 16 '21 at 17:05