Is it possible to get good FPS using Raspberry Pi camera v4l2 in c++?

Question

I'm trying to stream video on a Raspberry Pi using the official V4L2 driver with the Raspberry Pi camera, from C++ on raspbian (2015-02 release), and I'm having low FPS issues.

Currently I'm just creating a window and copying the buffer to the screen (which takes about 30ms) whereas the select() takes about 140ms (for a total of 5-6 fps). I also tried sleeping for 100ms and it decreases the select() time by a similar amount (resulting in the same fps). CPU load is about 5-15%.

I also tried changing the driver fps from console (or system()) but it only works downwards (for example, if I set the driver fps to 1fps, I'll get 1fps but if I set it to 90fps I still get 5-6fps, even though the driver confirms setting it to 90fps). Also, when querying FPS modes for the used resolution I get 90fps.

I included the parts of the code related to V4L2 (code omitted between different parts) :

//////////////////
// Open device
//////////////////
mFD = open(mDevName, O_RDWR | O_NONBLOCK, 0);
if (mFD == -1) ErrnoExit("Open device failed");

//////////////////
// Setup format
//////////////////
struct v4l2_format fmt;
memset(&fmt, 0, sizeof(fmt));
fmt.fmt.pix.pixelformat = V4L2_PIX_FMT_YUYV;
fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
Xioctl(VIDIOC_G_FMT, &fmt);
mImgWidth = fmt.fmt.pix.width;
mImgHeight = fmt.fmt.pix.height;
cout << "width=" << mImgWidth << " height=" << mImgHeight << "\nbytesperline=" << fmt.fmt.pix.bytesperline << " sizeimage=" << fmt.fmt.pix.sizeimage << "\n";
// For some reason querying the format always sets pixelformat to JPEG
//  no matter the input, so set it back to YUYV
fmt.fmt.pix.pixelformat = V4L2_PIX_FMT_YUYV;
if (Xioctl(VIDIOC_S_FMT, &fmt) == -1)
{
    cout << "Set video format failed : " << strerror(errno) << "\n";
}

//////////////////
// Setup streaming
//////////////////
struct v4l2_requestbuffers req;

memset(&req, 0, sizeof(req));

req.count = 20;
req.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
req.memory = V4L2_MEMORY_MMAP;

if (-1 == Xioctl(VIDIOC_REQBUFS, &req))
{
    ErrnoExit("Reqbufs");
}
if (req.count < 2)
    throw "Not enough buffer memory !";
mNBuffers = req.count;
mBuffers = new CBuffer[mNBuffers];
if (!mBuffers) throw "Out of memory !";

for (unsigned int i = 0; i < mNBuffers; i++)
{
    struct v4l2_buffer buf;
    memset(&buf, 0, sizeof(buf));
    buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    buf.memory = V4L2_MEMORY_MMAP;

    buf.index = i;

    if (-1 == Xioctl(VIDIOC_QUERYBUF, &buf))
        ErrnoExit("Querybuf");

    mBuffers[i].mLength = buf.length;
    mBuffers[i].pStart = mmap(NULL, buf.length, PROT_READ | PROT_WRITE, MAP_SHARED, mFD, buf.m.offset);

    if (mBuffers[i].pStart == MAP_FAILED)
        ErrnoExit("mmap");
}

//////////////////
// Start streaming
//////////////////
unsigned int i;
enum v4l2_buf_type type;
struct v4l2_buffer buf;

for (i = 0; i < mNBuffers; i++)
{
    memset(&buf, 0, sizeof(buf));

    buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    buf.memory = V4L2_MEMORY_MMAP;
    buf.index = i;

    if (-1 == Xioctl(VIDIOC_QBUF, &buf))
        ErrnoExit("QBUF");
}
type = V4L2_BUF_TYPE_VIDEO_CAPTURE;

if (-1==Xioctl(VIDIOC_STREAMON, &type))
    ErrnoExit("STREAMON");

And the last two parts in the main loop :

//////////////////
// Get frame
//////////////////
FD_ZERO(&fds);
    FD_SET(mFD, &fds);
    tv.tv_sec = 3;
    tv.tv_usec = 0;

    struct timespec t0, t1;

    clock_gettime(CLOCK_REALTIME, &t0);

    // This line takes about 140ms which I don't get
    r = select(mFD + 1, &fds, NULL, NULL, &tv);

    clock_gettime(CLOCK_REALTIME, &t1);

    cout << "select time : " << ((float)(t1.tv_sec - t0.tv_sec))*1000.0f + ((float)(t1.tv_nsec - t0.tv_nsec))/1000000.0f << "\n";

    if (-1 == r)
    {
        if (EINTR == errno)
            continue;
        ErrnoExit("select");
    }

    if (r == 0)
        throw "Select timeout\n";

    // Read the frame
    //~ struct v4l2_buffer buf;
    memset(&mCurBuf, 0, sizeof(mCurBuf));
    mCurBuf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    mCurBuf.memory = V4L2_MEMORY_MMAP;

    // DQBUF about 2ms
    if (-1 == Xioctl(VIDIOC_DQBUF, &mCurBuf))
    {
        if (errno == EAGAIN) continue;
        ErrnoExit("DQBUF");
    }

    clock_gettime(CLOCK_REALTIME, &mCaptureTime);

    // Manage frame in mBuffers[buf.index]
    mCurBufIndex = mCurBuf.index;

    break;
}

//////////////////
// Release frame
//////////////////
if (-1 == Xioctl(VIDIOC_QBUF, &mCurBuf))
    ErrnoExit("VIDIOC_QBUF during mainloop");

Although `select` seems a little too slow, why are you using it anyway? Why do you need a timeout (which adds more computation)? — edmz, Apr 17 '15 at 17:57

cww · Answer 1 · 2015-04-23T01:03:18.640

I have been looking at the various methods of using the picamera and I'm hardly an expert, but it would seem that the default camera settings are what's holding you back. There are many modes and switches. I don't know if they are exposed through ioctls or how yet, I just started. But I had to use a program called v4l-ctl to get things ready for the mode I wanted. A deep look at that source and some code lifting should let you achieve greatness. Oh, and I doubt the select call is an issue, it's simply waiting on the descriptor which is slow to become readable. Depending on mode, etc. there can be a mandatory wait for autoexposure, etc. Edit: I meant to say "a default setting" as you've changed some. There are also rules not codified in the driver.

Thanks for the answer. I didn't check the answers because I'm now using raspicam, which is a mmal wrapper and offers better performance. I don't want to dive into mmal itself (not a worthy investment for me!) so the intermediate layer offered by raspicam is good. — maxbc, Jun 08 '15 at 14:09

score -1 · Answer 2 · answered Oct 23 '16 at 19:01

The pixelformat matters. I encountered the similar low-fps problem, and I spent some time testing using my program in Go and C++ using V4L2 API. What I found is, Rpi Cam Module has good accelaration with H.264/MJPG pixelformat. I can easily get 60fps at 640*480, same as non-compressed formats like YUYV/RGB. However JPEG runs very slow. I can only get 4fps even at 320*240. And I also found the current is higher (>700mA) with JPEG compare to 500mA with H.264/MJPG.

Is it possible to get good FPS using Raspberry Pi camera v4l2 in c++?

2 Answers2