How to grab video frames fast?

Question

For my project I want to extract frames from a video to make thumbnails. However every method i find is super slow going through the entere video. Here is what I have tried:

with iio.imopen(v.file.path, "r") as vObj:
    metadata = iio.immeta(v.file.path, exclude_applied=False)
    frame_num = int(metadata['fps']*metadata['duration']-metadata['fps'])
    for i in range(10):
       x = int((frame_num/100)*(i*10))
       frame = vObj.read(index=x)
       path = v.get_thumbnail_path(index=i)
       os.makedirs(os.path.dirname(path), exist_ok=True)
       iio.imwrite(path, frame)
       logger.info('Written video thumbnail: {}'.format(path))

For a long video that takes extremely long. I know videos are compressed over multiple frames, however if I just manually open a video and jump to a point it also does not require to go through the video from first to last frame.

I don't care about specific frames, just roughly 10%, so sticking to keyframes is fine, if it makes it faster.

How to grab a frame every 10% of the video quickly?

Thank you.

Answer 1

The way you are approaching it is correct, and a current PR of mine (#939) will make the performance of calling read multiple times competitive.

Small benchmark:

import imageio.v3 as iio
import numpy as np
from timeit import Timer

def bench():
    with iio.imopen("cam1.mp4", "r", plugin="pyav") as file:
        n_frames = file.properties().shape[0]
        read_indices = np.linspace(0, n_frames-1, 10, dtype=int)
        for count, idx in enumerate(read_indices):
            frame = file.read(index=idx)
            iio.imwrite(f"thumbs/thumbnail_{count}.jpg", frame)

best = min(Timer("bench()", globals=globals()).repeat(5, number=1))
print(f"Best time: {best:.3f}")

Current (v2.25.0):   Best time: 2.134
Future (after #939): Best time: 0.924

The above benchmark uses a small, publically available video. The real-world gain will depend on the specific video being processed (and how keyframes are laid out within it). For example, in a longer video (several minutes, not publically available) you will notice a real difference:

Current (v2.25.0):   Best time: 42.952
Future (after #939): Best time: 1.687

This could be even faster for shorter GOP sizes, but that requires you to have control over how the video is produced and comes at the expense of increased file-size...

I don't care about specific frames, just roughly 10%, so sticking to keyframes is fine, if it makes it faster.

Reading keyframes isn't a reliable approach here unless you can make some assumptions/assertions about the types of videos you are operating on. Many videos aim for a GOP length of 250 frames but will have shorter/dynamic lengths based on how dynamic the content being encoded is. You can't generally know where keyframes are in advance, so you may end up with arbitrarily skewed results.

For example, a relatively slow-changing video recorded for 10 seconds at 25FPS (with a realized GOP of 250 frames) will have exactly 1 keyframe (the first one), and seeking to the closest keyframe anywhere in the video will always yield the first frame. Likely this isn't what you have in mind.