I recently followed this example (https://github.com/FFmpeg/FFmpeg/blob/master/doc/examples/vaapi_encode.c) to encode an AVFrame using NVENC (by swapping out AV_PIX_FMT_VAAPI with AV_PIX_FMT_CUDA and h264_vaapi with h264_nvence, among other things). I am now trying to decode the packet that I encoded and converting it to AV_PIX_FMT_YUV420P as such:
AVFrame *frame;
AVCodecContext *context;
AVCodec *codec;
avcodec_register_all();
codec = avcodec_find_decoder(AV_CODEC_ID_H264);
// allocate and set ffmpeg context
context = avcodec_alloc_context3(decoder->codec);
// open capture decoder
avcodec_open2(context, codec, NULL);
// alloc frame format
frame = (AVFrame *) av_frame_alloc();
frame->format = AV_PIX_FMT_CUDA;
// Receive AVPacket here ...
int success;
avcodec_decode_video2(context, frame, &success, &packet);
// Convert to YUV420P
struct SwsContext *sws_ctx = NULL;
sws_ctx = sws_getContext(1920, 1080,
AV_PIX_FMT_YUV420P, 1920, 1080,
AV_PIX_FMT_YUV420P,
SWS_BILINEAR,
NULL,
NULL,
NULL);
I am not able to pass AV_PIX_FMT_CUDA into sws_ctx
because it is not supported. Most examples I've found online say to follow this libav example (https://github.com/FFmpeg/FFmpeg/blob/master/doc/examples/hw_decode.c). The problem here is that I don't have an NVIDIA GPU on my decoding computer, so I can't create a hardware device as they do in this example. Does anyone have any suggestions on how to convert an AVFrame encoded to a packet using "h264_nvenc" to an AVFrame with format YUV420P? Thanks!