How to let FFMPEG fetch frames from OpenCV and stream them to HTTP server

Question

There is a camera that shoots at 20 frame per second. each frame is 4000x3000 pixel. The frames are sent to a software that contain openCV in it. OpenCV resizes the freames to 1920x1080 then they must be sent to FFMPEG to be encoded to H264 or H265 using Nvidia Nvenc. The encoded video then got steamed HTTP to a maximum of 10 devices.

The infrastructure is crazy good (10 GB Lan) with state of the art switchers, routers etc...

Right now, i can get 90 FPS when encoding the images from an Nvme SSD. this means that the required encoding speed is achieved.

The question is how to get the images from OpenCV to FFMPEG ? the stream will be watched on a webapp that was made using MERN stack (assuming that this is relevant).

Do you want to encode `cv::Mat` of `cv::cuda::GpuMat` frames? — zteffi, Aug 21 '20 at 14:25
@Zteffi I want to use cv::cuda::GPuMat if possible, it should be faster correct ? from my understanding, this will get the data directly to the GPU memory — Entropy, Aug 22 '20 at 16:46
@szatmary I need more details please, can you give me a link to a good source — Entropy, Aug 22 '20 at 16:47
Do you also need to remux audio or are you working with video only? — zteffi, Aug 22 '20 at 22:19

zteffi · Accepted Answer · 2021-02-21T21:44:58.133

For cv::Mat you have cv::VideoWriter. If you wish to use FFMpeg, assuming Mat is continuous, which can be enforced:

if (! mat.isContinuous())
{ 
    mat = mat.clone();
}

you can simply feed mat.data into sws_scale

sws_scale(videoSampler, mat.data, stride, 0, mat.rows, videoFrame->data, videoFrame->linesize);

or directly into AVFrame

For cv::cuda::GpuMat, VideoWriter implementation is not available, but you can use NVIDIA Video Codec SDK and similarly feed cv::cuda::GpuMat::data into NvEncoderCuda, just make sure your GpuMat has 4 channels (BGRA):

NV_ENC_BUFFER_FORMAT eFormat = NV_ENC_BUFFER_FORMAT_ABGR;
std::unique_ptr<NvEncoderCuda> pEnc(new NvEncoderCuda(cuContext, nWidth, nHeight, eFormat));

...
cv::cuda::cvtColor(srcIn, srcIn, cv::ColorConversionCodes::COLOR_BG2BGRA);
NvEncoderCuda::CopyToDeviceFrame(cuContext, srcIn.data, 0, (CUdeviceptr)encoderInputFrame->inputPtr,
            (int)encoderInputFrame->pitch,
            pEnc->GetEncodeWidth(),
            pEnc->GetEncodeHeight(),
            CU_MEMORYTYPE_HOST,
            encoderInputFrame->bufferFormat,
            encoderInputFrame->chromaOffsets,
            encoderInputFrame->numChromaPlanes);

Here's my complete sample of using GpuMat with NVIDIA Video Codec SDK

How to let FFMPEG fetch frames from OpenCV and stream them to HTTP server

1 Answers1