I have an app that broadcasts the screen of the device. I am using OpenTok library to send the frames. Their library does not handle the compressing process so every time I send a frame using this code to consume Buffer:
self.videoCaptureConsumer?.consumeImageBuffer(pixelBuffer, orientation:sample.orientation.openTokOrientation,timestamp:CMSampleBufferGetPresentatioTimeStamp(sample) ,metadata:nil)
the Broadcast extension crashes because of the memory limitation(50MB).
I have been searching in SO as well as repos in GitHub finally I ended up using a CPU image processing using Accelerate. I created an extension of CVPixelBuffer to resize the buffer. Here is the extension
I can resize the CVPixelBuffer and then send the new buffer to the OpenTok library. But the problem is since it is done by CPU, on iPhones X and below, the broadcast extension is stopped by the high CPU usage by the system.
So, I thought I have to find a way to compress-resize the buffer in a faster and memory-safer way with GPU acceleration.
Then I ended up checking Telegram's iOS app. I discovered the Telegram`s Broadcast Extension which actually works like a charm and it is what I need but I lack information on how it works since it uses C libraries.
My question is how I can compress-resize the CVPixelBuffer in a similar way that Telegram does but at least written with Swift language without passing memory limit and using GPU acceleration?
