5

I have a scientific application which captures a video4Linux video stream. It's crucial that we capture each frame and no one gets lost. Unfortunately frames are missing here and there and I don't know why.

To detect dropped frames I compare the v4l2_buffer's sequence number with my own counter directly after reading a frame:

void detectDroppedFrame(v4l2_buffer* buffer) {
        _frameCounter++;

        auto isLastFrame = buffer->sequence == 0 && _frameCounter > 1;
        if (!isLastFrame && _frameCounter != buffer->sequence+1)
        {
                std::cout << "\n####### WARNING! Missing frame detected!" << std::endl;               
                _frameCounter = buffer->sequence+1; // re-sync our counter with correct frame number from driver.
        }
}

My running 1-file example gist can be found at github (based on official V4L2 capture example): https://gist.github.com/SebastianMartens/7d63f8300a0bcf0c7072a674b3ea4817

Tested with webcam on Ubuntu 18.04 virtual machine on notebook-hardware (uvcvideo driver) as well as with CSI camera on our embedded hardware running ubuntu 18.04 natively. Frames are not processed and buffers seems to be grabbed fast enough (buffer status checked with VIDIOC_QUERYBUF which shows that all buffers are in the driver's incoming queue and V4L2_BUF_FLAG_DONE flag is not set). I have lost frames with MMAP as well as with UserPtr method. Also it seems to be independent of pixelformat, image size and framerate!

To me it looks like if the camera/v4l2 driver is not able to fill available buffers fast enough but also increasing the file descriptor priority with VIDIOC_S_PRIORITY command does not help (still likely to be a thread scheduling problem?).

=> What are possible reasons why V4L2 does not forward frames (does not put them into it's outgoing queue)? => Is my method to detect lost frames correct? Are there other options or tools for that?

Sebastian
  • 173
  • 10
  • hi, I think you have a bug at line 382. When ussing MMAP, the buffers are allocated by the driver and userspace should use mmap to map the memory to his mamoryspace. In your code the buffers are allocated in userspace instead of mmap. I think you better test the camera with the v4l-utils package which is written by the v4l2 developers – dafnahaktana Apr 10 '20 at 06:45
  • Thx. But from my understanding the "buffers" in ln382 are only holding adress and length of buffers, the allocation and mmap happens down below (ln 403). Do you know how can I leverage v4l2-utils (or any other tool) to test for lost frames? – Sebastian Apr 10 '20 at 07:15
  • right, my mistake. In the `v4l-utils` you can stream with --verbose option, then it shows the sequence of each buffer, or you can patch the source code and add a check – dafnahaktana Apr 10 '20 at 12:10
  • Issue reproduces with V4l2: `v4l2-ctl --stream-mmap --stream-count=1000 --verbose 2>&1 >/dev/null | grep Sequence` shows a list of sequence numbers. The last count will tell how many frames were lost. – Sebastian Apr 14 '20 at 06:28

1 Answers1

2

I had a similar problem when using the bttv driver. All attempts to capture at full resolution would result in dropped frames (usually around 10% of frames, often in bursts). Capturing at half resolution worked fine.

The solution that I found, though far from ideal, was to apply a load to the linux scheduler. Here is an example, using the "tvtime" program to do the capture:

#! /bin/bash
# apply a load in the background
while true; do /bin/true; done &> /dev/null &
# do the video capture
/usr/bin/tvtime "$@"
# kill the loop running in the background
pkill -P $$

This creates a loop that repeatedly runs /bin/true in the background. Almost any executable will do (I used /bin/date at first). The loop will create a heavy load, but with a multi-core system there is still plenty of headroom for other tasks.

This obviously isn't an ideal solution, but for what it's worth, it allows me to capture full-resolution video with no dropped frames. I have little desire to poke around in the driver/kernel code to find a better solution.

FYI, here are the details of my system:

OS:  Ubuntu 20.04, kernel 5.4.0-42
MB:  Gigabyte AB350M-D3H
CPU: AMD Ryzen 5 2400G
GPU: AMD Raven Ridge

Driver name      : bttv
Card type        : BT878 video (Hauppauge (bt878))
Bus info         : PCI:0000:06:00.0
Driver version   : 5.4.44
Capabilities     : 0x85250015
dgobbi
  • 491
  • 4
  • 6