Modifing and displaying QVideoFrames obtained in QAbstractVideoSurface

Question

I have very a simply application written in QT in which I want to display a movie by using QMediaPlayer, but before I will display any frame I would like to detect on it some objects and mark them by drawing a rectangle over it.

I've read in http://doc.qt.io/qt-5/videooverview.html that I can access each frame by subclassing QAbstractVideoSurface and so I dit it.

class VideoSurface : public QAbstractVideoSurface {
    Q_OBJECT

    bool present(const QVideoFrame &frame) override {
        if (surfaceFormat().pixelFormat() != frame.pixelFormat()
                || surfaceFormat().frameSize() != frame.size()) {
            setError(IncorrectFormatError);
            stop();

            return false;
        } else {
            currentFrame = frame;

            return true;
        }
    }

    ...
}

Now, I am receiving in this member function frames that I want to modify by drawing on it rectangles in places where I detected objects and then I would like to display them on the screen (preferably on some widget).

How can I do this?

1. Should my VideoSurface class contain QWidget as a member? or should I subclass QWidget which will contain VideoSurface?

2. In both cases, how can I display this frame? Should I first convert it to QImage and then display (it would be convinien for me, because my detection system is working with QImage, but would it be efficient)? I know that I can't paint from outside a paint event, so I can't paint in present function, so where exactly should be this painting function and how I can call it?

3. Where should I detect those object and modify frame? In present function, or in drawing function?

Answer 1

1. This is up to you and depends on how you prefer to structure your classes. I would prefer to have a separate widget that holds a pointer to your VideoSurface and draws the data that is returned by some member function of VideoSurface (depends on your solution in 2.)

2. a) QImage is efficient enough for some purposes and if you are already using it in your detection code, then you already have everything in memory and can work on that. As with all performance-related worries: Test and see if the performance is good enough for you. If it is not, you probably also have to consider doing the detection in a different way. I have worked on a project where we processed QImages converted from a similar VideoSurface on a camera stream on mobile devices (for relatively low-resolution images), and the performance was good enough that we haven't yet bothered to use other techniques. The source code for the VideoSurface class in that project (Neuronify) is hosted here. b) Your present() function could emit a signal that you can connect to from other objects that fetch the latest data from the VideoSurface and keeps it until their paint-function is called. Or you could apply the data directly to some widget that accepts image data. See Use of QAbstractVideoSurface for an example of that.

3. Again, that is up to you :) However, if you need to improve performance at some point you might want to do that work on a different thread to keep your GUI from locking up while you are processing the data. And if you do that, you need to decide whether you have to process every frame or if some frames can skip processing to improve the FPS of the playback. In the latter case, you should probably not do it in the present() function, as that is likely going to keep the media player from being able to feed you more frames while you are processing old frames.