FFmpeg hangs on buffer fill c++

Question

I'm processing a bunch of frames from an RTSP stream using ffmpeg. I end up doing a lot of processing on these frames, which means that I'm not always pulling in real-time. If the buffer gets full, the process hangs. I'm wondering if one of the following solutions is feasible/fixes the problem, and if so, how I would implement it using the ffmpeg libraries:

1) Is there a way to clear the buffer if I ever reach a point where it's hanging? (I can determine when it's hung, I just don't know what to do about it).

2) Is there a way to make the buffer overwrite the old data, and just always read the most recent data? It doesn't matter to me if I lose frames.

3) I've already discovered that I can make the buffer arbtrarily large with: av_dict_set(&avd, "buffer_size", "655360", 0);. This could be a solution, but I don't know how large/small it needs to be, because I don't know how long the stream will post video for?

4) Is this just a bug that I need to bring up with the ffmpeg people?

5) Something else I haven't considered?

while(av_read_frame(context, &(packet)) >= 0 && fcount < fps*SECONDS) {
    clock_t start, end;
    int ret = avcodec_send_packet(codec_context, packet);
    if(!(packet->stream_index == video_stream_index)) {
      continue;
    }

    if (ret == AVERROR(EAGAIN) || ret == AVERROR(EINVAL)) {
      continue;
    } else if (ret < 0) {
      cerr << "Error while decoding frame " << fcount << endl;
      exit(1);
    }

    ret = avcodec_receive_frame(codec_context, frame);
    if (ret == AVERROR(EAGAIN) || ret == AVERROR(EINVAL)) {
      continue;
    } else if (ret < 0) {
      cerr << "Error while decoding frame " << fcount << endl;
      exit(1);
    }

    sws_scale(img_convert_ctx, frame->data, frame->linesize, 0,
              codec_context->height, picture_rgb->data, picture_rgb->linesize);

    if(!frame) {
      cerr << "Could not allocate video frame" << endl;
      exit(1);
    }

    if(codec_context == NULL) {
      cerr << "Cannot initialize the conversion context!" << endl;
      exit(1);
    }

    // Do something with the frame here

    fcount++;
    av_packet_unref(&(packet));

}

I have added the code that causes the program to hang.

Answer 1

You could try you hand at multithreading. Each frame gets processed by a separate thread (use a threadpool).

If you require a sequential finishing order you will have to use some structure (queue?) to bring order in unordered threads finishing times.

Answer 2

Buffer size is a socket option, and under the hood ffmpeg may block on their socket recv call if the buffer is full. After looking at the ffmpeg code, it appears they have a FIFO size option for a circular buffer on pure UDP streams, but this is disabled for RTP streams.

To ensure the buffer never gets full I would run all pkt recv and frame decode operations in one thread and feed it to my own thread-safe circular buffer. A raspi should be able to keep up with decoding, but if not I would look into hardware assisted decoding. The point is, make sure your demuxing and decoding are maintaining realtime speed. You can handle all your resizing, processing, and frame writing in other threads and set the circular buffer size accordingly to handle overflow.

It appears you are trying to write raw frames in realtime so a word of warning: obviously, fps and frame size plays a role in the speed of everything, but I suspect you will see a lot of dropped frames.

Answer 3

This should be a hint not a direct solution for your problem. Buffering while converting stream to frames with ffmpeg

What you should try is to record the stream and cache it to the storage, then you convert it. What you also can try is caching the buffer to the disk and refeed the ffmpeg process.

A hacky solution would be to convert the RTSP Stream to UDP which is in fact easy af and then use the ffmpeg internal buffer options available for UDP which are disabled for RTSP