2

All the implementation i saw in optical flow in opencv uses video as an array of frames and then implement optical flow on each image. That involves slicing the image into NxN block and searching for velocity vector.

Although motion vector in video codec is misleading and it does not necessarily contain motion information, why don't we use it to check which block likely has motion and then run optical flow on those blocks ? Shouldn't that fasten the process ?

Marcus Müller
  • 34,677
  • 4
  • 53
  • 94
cowboysaif
  • 241
  • 2
  • 9

3 Answers3

3

Motion vector calculated for video encoding is not 'True Motion Vector'. The calculation targets to find best matching block to achieve highest compression. It doesn't targets the best motion estimation. That's why it can't be readily used for motion estimation purpose.

However, the motion-vectors from decoder can be used in some way to make motion estimation faster and better process. e.g.

  1. As a seed to your algorithm of motion estimation.
  2. Try filter like median (depending on data) to remove outlier motion vectors and use it for better motion estimation.
  3. One of above step can be used as first step in a two-step motion estimation algo.
Pervez Alam
  • 1,246
  • 10
  • 20
2

OpenCV is a universal image processing framework. It takes in frames, not compressed video, for its algorithms.

You can certainly write a video decoder that also hands out info about displacement from the codec to openCV – but that will be very codec-specific and thus isn't in scope of openCV itself.

Marcus Müller
  • 34,677
  • 4
  • 53
  • 94
  • thank you. In a very highly theoretical sense, should motion vector data speed up the process of the optical flow, discarding the codec and opencv related implementation ? I mean, motion vector in video codec IS velocity vector in optical flow, albeit highly optimized and uses different algorithm for achieving better compression. – cowboysaif Aug 20 '17 at 22:04
  • probably? It's impossible to say without looking at the specific codec, and figuring out how much overhead it is to get the motion data out of the decoder – which probably involves simply implementing what we're talking about. Have a look at VLC's source code. IIRC, VLC can at least display motion vectors as small arrows. Chances are, you can adopt that mechanism. – Marcus Müller Aug 20 '17 at 22:09
1

I distinctly remember having read academic work done on the use of "motion" vectors embedded in h.264 and similar codecs for optical flow/vision motion analysis. The easiest approach to do so, is for iterative estimations (such as Horn and Schunk). One can simply bootstrap the iterative algorithm with the vectors from the decoder.

This will not improve the estimation accuracy, at best I would guess it could speed up the rate of convergence, allowing for earlier stopping.

Not all h.264 encoding is of the same quality either. Especially from real-time systems with limited hardware. There are likely cases where the motion vectors will be outright awful, and actually be a detriment to the flow estimations instead of helping. I am thinking for example of a low-end IP camera that is designed for mostly static scenes, that is moving in varying lighting etc.

Stefan Karlsson
  • 1,092
  • 9
  • 21