How do you access the frame number inside AVAsynchronousCIImageFilteringRequest?

Question

When using a CIFilter with an AVVideoComposition, how can I access the current frame number in the AVAsynchronousCIImageFilteringRequest callback?

The best I was able to do is estimate the frame number from the time:

^(AVAsynchronousCIImageFilteringRequest * _Nonnull request) {
    double seconds = CMTimeGetSeconds(request.compositionTime);
    double fps = [[_avAsset tracksWithMediaType:AVMediaTypeVideo] firstObject].nominalFrameRate;
    int frame = round(seconds * fps);

    // (Calculate filter parameters based on frame number)
}

But this isn't accurate enough for me. Is there a way to access the frame number?

score 3 · Accepted Answer · edited Aug 06 '18 at 07:34

The key is to set the frameDuration on the AVMutableVideoComposition to match that of the underlying AVAssetTrack. if you do this, then the values of compositionTime will be even multiples of minFrameDuration and there won't be any rounding issues.

AVMutableVideoComposition *vidComp =
    [AVMutableVideoComposition videoCompositionWithAsset:self.avAsset
                            applyingCIFiltersWithHandler:
        ^(AVAsynchronousCIImageFilteringRequest * _Nonnull request) {
            CMTime frame = self.avAssetTrack.minFrameDuration;
            int frameNum = (request.compositionTime.value * frame.timescale) /
                           (request.compositionTime.timescale * frame.value);

            // (Calculate filter parameters based on frame number)
        }];

vidComp.frameDuration = self.avAssetTrack.minFrameDuration;

In my case of a ~60 fps MP4 movie, avAssetTrack.minFrameDuration = (1001/60000), but vidComp.frameDuration = (1501/90000) before I update it. This resulted in slow drift of 'compositionTime' in the callback, which resulted in occasional repeated frames.

I'm curious how you're setting vidComp.frameDuration since it's read only. — Ivan Lesko, Aug 24 '16 at 19:19
It's not read-only in an `AVMutableVideoComposition` (as opposed to an `AVVideoComposition`). — Matthew Self, Aug 25 '16 at 19:33

Rhythmic Fistman · Answer 2 · 2016-06-23T22:25:16.493

0

How about (a hacky)

__block int frameNum = 0;

AVMutableVideoComposition *vidComp = 
    [AVMutableVideoComposition videoCompositionWithAsset:self.avAsset
        applyingCIFiltersWithHandler:^(AVAsynchronousCIImageFilteringRequest * request) {

        // (Calculate filter parameters based on frame number)

        frameNum++;
    }];

?

There are two assumptions here:

your handler is called in the same order as the frames
your handler is not called by more than one thread at a time

I don't know if these are safe assumptions or not.

p.s. frame rates are quite often variable, so min, max and frame duration can be misleading.

UPDATE the AVComposition from the question is intended for seekable playback in an AVPlayer which violates assumption 1.

edited Jun 23 '16 at 22:25

answered Jun 22 '16 at 13:51

Rhythmic Fistman

34,352
5
87
159

The video is being played through an AVPlayerView, so the user can seek anywhere they want in the video, or play FF or reverse, and I need to know the frame number even in these cases. – Matthew Self Jun 23 '16 at 16:07
As far as I can tell, if you set `vidComp.frameDuration`, then `compositionTime` in the callback will always be an exact multiple of that even if the frame rate of the videos being composed are variable. – Matthew Self Jun 23 '16 at 16:10
Oh. I should add that my answer will definitely not work for arbitrary playback. That's a cool trick with `frameDuration`/`compositionTime`. Can you accept your answer yet? – Rhythmic Fistman Jun 23 '16 at 22:16

How do you access the frame number inside AVAsynchronousCIImageFilteringRequest?

2 Answers2