Qt: getting a thumbnail for a video

Question

Is there a better way to get a thumbnail for a video? OS is primarily Linux, but hopefully there's a cross platform way to do it. This is what I have right now:

from PySide6 import QtMultimedia as qtm
from PySide6 import QtMultimediaWidgets as qtmw
from PySide6 import QtCore as qtc

app = qtw.QApplication()

thumbnail_file = "video.mp4"
loop = qtc.QEventLoop()
widget = qtmw.QVideoWidget()
widget.setVisible(False)
media_player = qtm.QMediaPlayer()
media_player.setVideoOutput(widget)
media_player.mediaStatusChanged.connect(loop.exit)
media_player.positionChanged.connect(loop.exit)
media_player.setSource(thumbnail_file)
loop.exec()
media_player.mediaStatusChanged.disconnect()
media_player.play()
if media_player.isSeekable():
    media_player.setPosition(media_player.duration() // 2)
loop.exec()
media_player.positionChanged.disconnect()
media_player.stop()
image = media_player.videoSink().videoFrame().toImage()
image.save('thumbnail.jpg')
app.exec()

This will be ran in a separate thread so the time is not really an issue, but it's still pretty convoluted.

Just a comment, it does not jump to the middle of the track as you would probably expect. You should probably call `media_player.play()` after `setPosition()`. At least it works for me correctly after this change. Otherwise, I juggled with the code and did not find any way to simplify it. You can however leave out line `app.exec()`. — HiFile.app - best file manager, Sep 01 '22 at 14:59
@HiFile.app-bestfilemanager -> "it works for me correctly after this change", Interesting, it works for me as expected as it is. — Neat, Sep 01 '22 at 15:07
@Neat note that, as the documentation [explains](https://doc.qt.io/qt-6/qmediaplayer.html#duration-prop), the duration "may not be available when initial playback begins", so you should always call `setPosition()` **if** the duration is > 0 or at least when `durationChanged` is emitted. Besides, if you need to run the above in a thread, you cannot use QVideoWidget. — musicamante, Sep 01 '22 at 15:17
Before that change, it always captured the very first video frame from the very beginning of the video, in my case. — HiFile.app - best file manager, Sep 01 '22 at 15:18
@musicamante That is very good point about the widget in thread. Without setting the widget, the algorithm does ot work (the video does not play). So the question is how to work around this constraint... — HiFile.app - best file manager, Sep 01 '22 at 15:26
@HiFile.app-bestfilemanager I don't have the latest Qt6 version and didn't test the new multimedia framework yet, but, if I get it correctly, it should be possible to create a QVideoSink subclass and just work with that after calling `setVideoSink()`. — musicamante, Sep 01 '22 at 15:32
@musicamante -> "if you need to run the above in a thread, you cannot use QVideoWidget", Do you mean to say I should not or that it's not possible? Because I'm running that whole block of code in a separate thread and it's working fine for me. — Neat, Sep 01 '22 at 15:37
@Neat **If** you don't need a main QApplication, then you can as long as QApplication *is* in that same thread in which you're going to create widgets. But, as said above, since you're not showing anything, there's little point in using QVideoWidget, and you should consider other aspects (like trying QVideoSink as suggested above). — musicamante, Sep 01 '22 at 16:17
@musicamante -> "QVideoSink", My apologies, I couldn't put two and two together to understand that you were suggesting replacing the widget with a `QVideoSink` with `setVideoSink()`, that works (without the need of a subclass). — Neat, Sep 01 '22 at 16:23
I can confirm, that instantiation of `QVideoSink` and setting it with `setVideoSink()` works fine. @musicamante is genius. — HiFile.app - best file manager, Sep 01 '22 at 19:44
This doesn't work on linux. As pointed out above, it only ever saves the first frame. However, recalling `play()` also makes no difference. The only reliable way to fix it is to connect the sink's `videoFrameChanged` signal to `loop.exit` after the `positionChanged` signal has fired, and then restart the loop. This ensures the player has advanced to the correct frame before saving the image. It *is* possible to simplify the code somewhat by calling `processEvents` in a while-loop. That will eliminate all the signal connections and loop exits, but the overall approach is essentially the same. — ekhumoro, Sep 02 '22 at 14:09
@ekhumoro thank you for the reply, is it necessary to wait for the `positionChanged` signal to fire before again waiting for `videoFrameChanged` signal to fire? Wouldn't `videoFrameChanged` alone serve the purpose of both? — Neat, Sep 03 '22 at 02:35
@Neat The official docs are somehow unclear at this point (remember that the QtMultimedia framework has changed in Qt6, it was reintroduced only since 6.2, and it's still in active development), but I believe that it's still possible that the `videoFrameChanged` signal *could* be emitted **before** `positionChanged` if for some reason the player is able to load the first frame before being able to change the position due to the fact that you should wait for the `durationChanged` signal in order to get the proper position (video files are *odd*). My suggestion is to wait and check for *both*. — musicamante, Sep 03 '22 at 03:31
@musicamante that is a possibility. I shall do just that then. — Neat, Sep 03 '22 at 03:33
@Neat If I may, since I believe you're trying to create some sort of video manager. Media playing/management is an incredibly complex subject: you have to deal with tons of different "formats" (as in "containers" of media, including images) and codecs, and while some of those formats were somehow "strict", variations of them have become "standards" (read about AVI/DivX), especially due to "universal players" (such as VLC) and the flexibility of network usage. I strongly suggest you to do some careful, patient and thorough research on the matter, including cross-platform/media-library support. — musicamante, Sep 03 '22 at 03:52
@musicamante appreciate the heads up! I do plan on doing as much research as I can on this topic. Right now, I'm just trying to get a working "prototype" to act as the base for further improvements. Though, would you happen to know if I can alleviate some/all of this burden of research by just making use of the `mpv` library on Python? — Neat, Sep 03 '22 at 04:02
@Neat Well, you can embed an mpv video in Qt, but that's not immediate, especially considering the direct interaction of mpv with user events. Be aware, though, that mpv is based on ffmpeg. I was actually thinking of suggesting you to do some research on ffmpeg too, as it's considered quite a standard nowadays. So, if you plan on using any of it (and especially if your main target is *nix platforms), you should start by patiently and thoroughly studying the ffmpeg documentation anyway. If you'll do it (and you'll have to, sooner or later), then... have fun, see you (hopefully) in 2023 ;-) — musicamante, Sep 03 '22 at 04:24
@musicamante -> "you should start by patiently and thoroughly studying the ffmpeg documentation" will do, thank you for all the help! — Neat, Sep 03 '22 at 04:33
@Neat I have posted an answer with an alternative solution which hopefully explains what the correct approach should be. — ekhumoro, Sep 03 '22 at 12:25

score 2 · Answer 1 · answered Sep 03 '22 at 12:22

There are many different approaches to this, since QMediaPlayer can undergo multiple state changes which can be monitored in a variety of ways. So the question of what is "best" probably comes down to what is most reliable/predictable on the target platform(s). I have only tested on Arch Linux using Qt-6.3.1 with a GStreamer-1.20.3 backend, but I think the solution presented below should work correctly with most other setups.

As it stands, the example code in the question doesn't work on Linux. As pointed out in the comments, this can be fixed by using the videoFrameChanged signal of the mediad-player's video-sink. (Unfortunately, at time of writing, these APIs are rather poorly documented). However, the example can be simplified and improved in various ways, so I have provided an alternative solution below.

During testing, I found that the durationChanged and postionChanged signals cannot be relied upon to give the relevant notifications at the appropriate time, so I have avoided using them. It's best to wait for the player to reach the buffered-state before setting the position, and then wait until a frame is received that can be verified as matching the requested position before saving the image. (I would also advise adding a suitable timeout in case the media-player gets stuck in an indeterminate state).

To illustrate the timing issues alluded to above, here is some sample output:

frame changed: 0
duration change: 13504
status change: b'LoadedMedia'
frame changed: 0
status change: b'BufferedMedia'
set position: 6752
frame changed: 6752
save: exit

frame changed: 0
frame changed: 0
duration change: 13504
status changed: b'LoadedMedia'
status changed: b'BufferedMedia'
set position: 6752
frame changed: 6752
save: exit

As you can see, the exact sequence of events isn't totally predictable. If the position is set before the buffered-state is reached, the output looks like this:

frame changed: 0
duration changed: 13504
status changed: LoadedMedia
set position: 0
frame changed: 0
status change: BufferedMedia
frame changed: 40
frame changed: 80
... # long list of changes
frame changed: 6720
frame changed: 6760
save: exit

Here is the demo script (which works with both PySide6 and PyQt6):

import os
from PySide6.QtCore import QCoreApplication, QTimer, QEventLoop, QUrl
from PySide6.QtMultimedia import QMediaPlayer, QVideoSink
# from PyQt6.QtCore import QCoreApplication, QTimer, QEventLoop, QUrl
# from PyQt6.QtMultimedia import QMediaPlayer, QVideoSink

def thumbnail(url):
    position = 0
    image = None
    loop = QEventLoop()
    QTimer.singleShot(15000, lambda: loop.exit(1))
    player = QMediaPlayer()
    player.setVideoSink(sink := QVideoSink())
    player.setSource(url)
    def handle_status(status):
        nonlocal position
        print('status changed:', status.name)
        # if status == QMediaPlayer.MediaStatus.LoadedMedia:
        if status == QMediaPlayer.MediaStatus.BufferedMedia:
            player.setPosition(position := player.duration() // 2)
            print('set position:', player.position())
    def handle_frame(frame):
        nonlocal image
        print('frame changed:', frame.startTime() // 1000)
        if (start := frame.startTime() // 1000) and start >= position:
            sink.videoFrameChanged.disconnect()
            image = frame.toImage()
            print('save: exit')
            loop.exit()
    player.mediaStatusChanged.connect(handle_status)
    sink.videoFrameChanged.connect(handle_frame)
    player.durationChanged.connect(
        lambda value: print('duration changed:', value))
    player.play()
    if loop.exec() == 1:
        print('ERROR: process timed out')
    return image

video_file = 'video.mp4'
thumbnail_file = 'thumbnail.jpg'

try:
    os.remove(thumbnail_file)
except OSError:
    pass

app = QCoreApplication(['Test'])

image = thumbnail(QUrl.fromLocalFile(video_file))
if image is not None:
    image.save(thumbnail_file)

Thanks for your insights about the issue. C++ coder here, but this helped me a lot with Qt 6.5. — Alvein, Feb 11 '23 at 05:24

Neat · Answer 2 · 2022-09-03T04:26:54.670

Answer reached thanks to the discussion in the comments of OP:

from PySide6 import QtWidgets as qtw
from PySide6 import QtMultimedia as qtm
from PySide6 import QtMultimediaWidgets as qtmw
from PySide6 import QtCore as qtc

app = qtw.QApplication()

thumbnail_file = "video.mp4"
loop = qtc.QEventLoop()
video_sink = qtm.QVideoSink()
media_player = qtm.QMediaPlayer()
media_player.setVideoSink(video_sink)
media_player.mediaStatusChanged.connect(loop.exit)
media_player.setSource(thumbnail_file)
loop.exec()
media_player.mediaStatusChanged.disconnect()
if media_player.isSeekable() and media_player.duration():
    media_player.positionChanged.connect(loop.exit)
    media_player.setPosition(media_player.duration() // 2)
    media_player.play()
    loop.exec()
    media_player.positionChanged.disconnect()
else:
    media_player.play()
video_sink.videoFrameChanged.connect(loop.exit)
loop.exec()
video_sink.videoFrameChanged.disconnect()
media_player.stop()
image = media_player.videoSink().videoFrame().toImage()
image.save('thumbnail.jpg')

score 0 · Answer 3 · answered Mar 31 '23 at 06:55

FFMPEG can generate thumbnails per sec or every N sec.

# generate thumbnails of 120 pixel width and its corresponding height 
# while keeping the size ratio every 10 sec.

command = "ffmpeg -y -i {} -vf fps=1/10,scale=120:-1 tmp/thumb/%d.png".format(self.path)
os.system(command)

If you want the first frame only,

command = "ffmpeg -y -ss 00:00:00 -i {} -frames:v 1 tmp/intro.png".format(self.path)
os.system(command)

Qt: getting a thumbnail for a video

3 Answers3

Linked