Display on screen using QAbstractVideoSurface

Question

I am trying to display camera picture on screen using a subclass of QAbstractVideoSurface, and I've no experience on this.

I'd appreciate if anyone can explain how to do it.

Answer 1

The QAbstractVideoSurface is an interface between the producer and consumer of the video frames. You only have two functions to implement to begin with:

1. supportedPixelFormats so that the producer can select an appropriate format for the QVideoFrame
2. present which is more generic wording for show\display this frame

Lets say you want to use a classic QWidget for the display. In this case, you may choose to use a QImage to draw on the widget.

First Qt is guaranteed to paint a QImage which is RGB24 (or BGR24) on most platforms. So

QList<QVideoFrame::PixelFormat> LabelBasedVideoSurface::supportedPixelFormats(
        QAbstractVideoBuffer::HandleType handleType) const
{
    if (handleType == QAbstractVideoBuffer::NoHandle) {
        return QList<QVideoFrame::PixelFormat>()
                << QVideoFrame::Format_RGB24;
    } else {
        return QList<QVideoFrame::PixelFormat>();
    }
}

Now to present the QVideoFrame, you map its data to a QImage, and paint the QImage to the widget. For simplicity I will use a QLabel, that I access directly (no signal no slot).

bool LabelBasedVideoSurface::present(const QVideoFrame &frame)
{
    if (notMyFormat(frame.pixelFormat())) {
        setError(IncorrectFormatError);
        return false;
    } else {

        QVideoFrame frametodraw(frame);

        if(!frametodraw.map(QAbstractVideoBuffer::ReadOnly))
        {
           setError(ResourceError);
           return false;
        } 

         //this is a shallow operation. it just refer the frame buffer
         QImage image(
                frametodraw.bits(),
                frametodraw.width(),
                frametodraw.height(),
                frametodraw.bytesPerLine(),
                QImage::Format_RGB444);

        mylabel->resize(image.size());

        //QPixmap::fromImage create a new buffer for the pixmap
        mylabel->setPixmap(QPixmap::fromImage(image));

        //we can release the data
        frametodraw.unmap();

        mylabel->update();

        return true;
    }
}

This example is obviously not optimal.

1. It doesnt take cash on the fact that a QVideoFrame might be stored in video memory, because we are drawing using a pixmap.
2. The conversion from image to pixmap is a unnecessary hit.

You can write your own widget, and implement a paintEvent for better performance. Also, you have several design liberties on how present() behave. For instance :

• Whether is a non blocking surface, ie the frame is already shown when present finishes. Above it will mean using mylabel->repaint() instead of mylabel->update()
• What happens when you cannot complete the presentation. You may want to draw a blank frame rather than returning an error who may stop the music.