6

I have an application which connects to an RTSP camera and processes some of the frames of video. Depending on the camera resolution and frame rate, I don't need to process all the frames and sometimes my processing takes a while. I've designed things so that when the frame is read, its passed off to a work queue for another thread to deal with. However, depending on system load/resolution/frame rate/network/file system/etc, I occasionally will find cases where the program doesn't keep up with the camera.

I've found that with ffmpeg(I'm using the latest git drop from mid october and running on windows) that being a couple seconds behind is fine and you keep getting the next frame, next frame, etc. However, once you get, say, 15-20 seconds behind that frames you get from ffmpeg occasionally have corruption. That is, what is returned as the next frame often has graphical glitches (streaking of the bottom of the frame, etc).

What I'd like to do is put in a check, somehow, to detect if I'm more than X frames behind the live stream and if so, flush the caches frames out and start fetching the latest/current frames.

My current snippet of my frame buffer reading thread (C++) :

while(runThread)
{
    av_init_packet(&(newPacket));

    int errorCheck = av_read_frame(context, &(newPacket));
    if (errorCheck < 0)
    {
        // error
    }
    else
    {

        int frameFinished = 0;
        int decodeCode = avcodec_decode_video2(ccontext, actualFrame, &frameFinished, &newPacket);

        if (decodeCode <0)
        {
            // error
        }
        else
        if (decodeCode == 0)
        {
            // no frame could be decompressed / decoded / etc
        }
        else
        if ((decodeCode > 0) && (frameFinished))
        {
            // do my processing / copy the frame off for later processing / etc
        }
        else
        {
            // decoded some data, but frame was not finished...
            // Save data and reconstitute the pieces somehow??
            // Given that we free the packet, I doubt there is any way to use this partial information
        }
        av_free_packet(&(newPacket));
    }
}

I've google'd and looked through the ffmpeg documents for some function I can call to flush things and enable me to catch up but I can't seem to find anything. This same sort of solution would be needed if you wanted to only occasionally monitor a video source(eg, if you only wanted to snag one frame per second or per minute). The only thing I could come up with is disconnecting from the camera and reconnecting. However, I still need a way to detect if the frames I am receiving are old.

Ideally, I'd be able to do something like this :

while(runThread)
{
    av_init_packet(&(newPacket));

    // Not a real function, but I'd like to do something like this
    if (av_check_frame_buffer_size(context) > 30_frames)
    {
        // flush frame buffer.
        av_frame_buffer_flush(context);
    }

    int errorCheck = av_read_frame(context, &(newPacket));

    ...
    }
}
LawfulEvil
  • 2,267
  • 24
  • 46
  • possibly losing UDP packets due to too full of buffers? I think I've heard of this before [ffmpeg smeared bottom, VLC ok] http://ffmpeg.zeranoe.com/forum/viewtopic.php?f=15&t=1000&p=4055&hilit=axis#p4055 but I'm not sure if anybody figured it out all the way...is this the same problem? – rogerdpack Nov 06 '14 at 17:50
  • Some minor issues were solved by switching to TCP. However, I did that after finding the suggestion on google. All issues described above are while using TCP. VLC doesn't use ffmpeg. This is a problem where ffmpeg assumes that you will want to read all the frames (?) and there doesn't seem to be a way (that I have found) to skip frames and read the current frame even though the last frame you read might have been 10 seconds ago. – LawfulEvil Nov 10 '14 at 18:24
  • maybe ask on the libav-user ML or ffmpeg-user ML, GL! see also https://trac.ffmpeg.org/ticket/1734 – rogerdpack Nov 10 '14 at 20:46
  • 3
    I've come up with two solutions to the problem depending on how infrequently you want to read frames. 1. Close the rtsp connection after reading the frame and re-establish each time you need a frame. Assuming you only want very infrequent frames. 2. Have your inner loop read all the frames and hang onto the latest one so when your main application needs its infrequent frame, there is one available. You do have the overhead/waste of reading all the frames. – LawfulEvil Nov 18 '14 at 13:48

0 Answers0