2

In my understanding (a very simplistic view), the inter prediction (motion estimation / compensation) of H264 standard first finds the best match block on the reference frame, and then it's encoded with motion vector (the effective new X and Y) and the residual (prediction vs reality).

But how does the decoder knows how to fill the old space where the predicted block was before? I'm supposing that the residual is calculated from its new position, on a block level, not a frame level.

Let's say the encoder decided to use inter prediction to the encode the following two images, it calculates where does the ball should be (its new position and the residual energy) but how does it fill the old space?

frame 0 frame 1

Leandro Moreira
  • 377
  • 3
  • 16

1 Answers1

3

Motion compensation is just optimization of frame encoding. If we talk about motion vectors, this is "block motion compensation", as defined in Wikipedia:

Block motion compensation divides up the current frame into non-overlapping blocks, and the motion compensation vector tells where those blocks come from (a common misconception is that the previous frame is divided up into non-overlapping blocks, and the motion compensation vectors tell where those blocks move to). The source blocks typically overlap in the source frame. Some video compression algorithms assemble the current frame out of pieces of several different previously-transmitted frames.

So, motion vector is not "move the block from old frame", it is (intra-) encoding of current frame's macroblocks, when some blocks are copied from older frame with some small shift (and some blocks can be copied several times from previous frame into current; and most block are copied with zero motion vector). In theory we can encode new frame by its macroblocks, but with help of motion compensation we have change to get lot of image information from previous frames and encode less. Parts of images which are not compensated are encoded with image macroblocks.

Example from other Wikipedia page from "Big Buck Bunny" free film:

Frame with motion vectors displayed, dots are zero vectors

Residual image after motion compensation to be encoded

There is good description of the encoding/decoding processes in H.264 but in russian: http://www.compression.ru/dv/course/compr_h264.pdf (from http://www.compression.ru/video/ site)

And description of motion compensation in English: http://inst.eecs.berkeley.edu/~ee290t/sp04/lectures/02-Motion_Compensation_girod.pdf (prediction error is encoded, and completely new image will be is almost fully mispredicted. Background image has low frequency of information and most probably will be motion compensated by some of nearby background block.)

osgx
  • 90,338
  • 53
  • 357
  • 513
  • Thank you so much :D, isn't inter the word you want to use here "it is (intra-) encoding of current frame's macroblocks,"? – Leandro Moreira Mar 04 '17 at 03:51
  • leandro, I did video of your two frames (http://trac.ffmpeg.org/wiki/Slideshow I+P), but macroblocks are not visible in it. Try to use ffmpeg with debugging as described in https://trac.ffmpeg.org/wiki/Debug/MacroblocksAndMotionVectors – osgx Mar 04 '17 at 08:27
  • 1
    in order to better understand this issue I did an experiment at https://github.com/leandromoreira/digital_video_introduction/blob/master/frame_difference_vs_motion_estimation_plus_residual.ipynb and soon I'll be tackling the temporal redundancy https://github.com/leandromoreira/digital_video_introduction#temporal-redundancy-inter-prediction , this is all WIP right now. – Leandro Moreira Mar 07 '17 at 17:03
  • So, "Generate debug video" with ffmpeg works for you? Did you change something inside ffmpeg, and which version was used? Interesting project https://github.com/leandromoreira/digital_video_introduction hope it wil have more info and examples. – osgx Mar 07 '17 at 19:37
  • Yes, it worked pretty well. I'm using a containerized version of ffmpeg (through docker), it's the same I use to run the examples in the project. – Leandro Moreira Mar 08 '17 at 04:30