2

What is the best way to detect motions of objects on video (exclude ego motion of camera) when the my camera moves?

I have real-time 50 FPS fullHD 1920x1080 video-stream. Camera has ego motion.

I.e. I must do:

  • estimate motion of my camera between 1st and 2nd frames (translation, scale and rotation)
  • compensate this movement of my camera - reversed move 2-nd frame
  • use gpu::MOG2_GPU to detect motions of objects - the difference between the two frames

What is the best way (the fastest and the best quality) to compensate movement of the my camera?

  1. phaseCorrelate() - get only translation and very slow
  2. cv::videostab - class designed to eliminate jitter and video stabilization
  3. for each frame calculate (gpu::SURF_GPU or goodFeaturesToTrack()) and estimateRigidTransform() - get affine transformation (translation, scale and rotation) but is very slow for HD real-time video
  4. for every 50 th frame (1 times per 1 sec) calculate (gpu::SURF_GPU or goodFeaturesToTrack()) and estimateRigidTransform()
  5. use approach 4, but with custom implementation of estimateRigidTransform() - use (calcOpticalFlowSF() or calcOpticalFlowFarneback() or DenseOpticalFlow::calc) instead of calcOpticalFlowPyrLK(): https://github.com/Itseez/opencv/blob/3942b1f36261b196a264eb35c996222848fe3c93/modules/video/src/lkpyramid.cpp#L1439

What is the best of these 5 approaches, and if this is 4, then what is the best of OpticalFlow-functions:

Alex
  • 12,578
  • 15
  • 99
  • 195
  • I always wanted to try dense optical flow to combine it with visual flow patterns analyzed by Gibson (psychologist) but didnt find the time/project for it :D – Micka Dec 05 '15 at 15:27
  • @Micka This is an important point, that we are about (visual flow patterns) James Gibson, but not about (Pattern Recognition novel) William Gibson :) James Gibson's (1966) bottom up processing, which is vs Gregory's (1970) top down processing: http://www.simplypsychology.org/perception-theories.html Do you think should try it to compensate ego motion? – Alex Dec 05 '15 at 16:04
  • 1
    probably it is quite slow, but I like it in theory... but it was really just a comment, no idea whether it will work. Good luck with the question, I like that you presented many candidates of how to solve the problem. – Micka Dec 05 '15 at 16:48
  • @Micka I'll try to use `gpu::PyrLKOpticalFlow::sparse()` or `gpu::PyrLKOpticalFlow::dense()`, then `gpu::meanShiftSegmentation()` to separate different flows by clear boundaries, and then for each segment build two `std::vector` with source coordinates (each 5 pixels) and destination coordinates (calc from source and flow from step_1) of points in this segment, and try to find affine transformation `Mat affine = cv::estimateRigidTransform(src_pts, dst_pts)`, and compare each affine transformation with each Gibson's visual flow patterns. Or is it possible to make somehow better? – Alex Dec 06 '15 at 19:30
  • The biggest segment of flow - must be a ego motion of camera. Others - movement of objects. – Alex Dec 06 '15 at 19:34
  • would be similar approach I would try but somehow compare the whole flow with gibson patterns. maybe with ransac so the object movements become outlier – Micka Dec 06 '15 at 19:38
  • @Micka I already tested something like this but with sparse flow. I use `Mat affine = estimateRigidTransform(frame1, frame2)`, then compensate ego motion `gpu::warpAffine(frame2, frame2_warped, affine, frame2.size(), WARP_INVERSE_MAP);`, and then detect motions of objects `gpu::MOG2_GPU` for `frame2_warped`. `estimateRigidTransform()` uses sparse flow https://github.com/Itseez/opencv/blob/3942b1f36261b196a264eb35c996222848fe3c93/modules/video/src/lkpyramid.cpp#L1439 and RANSAC https://github.com/Itseez/opencv/blob/3942b1f36261b196a264eb35c996222848fe3c93/modules/video/src/lkpyramid.cpp#L1467 – Alex Dec 07 '15 at 20:28
  • @Micka `affine` transformation (translation, scale and rotation) well describes Gibson's optical flow patterns, and RANSAC well filter out outliers (random noise). But if we have 2 big movements: 1-st background (ego motion), 2-nd hood of the car, or moving my hand close to the camera, or moving big object, then `estimateRigidTransform()` with RANSAC calculates **average transformation between these 2 big movements**- bad compensation of ego motion and bad background subtraction. Then I want to have a way to manualy customize threshold of the boundary separation between the massive movements. – Alex Dec 07 '15 at 20:40
  • hi there, just curious; what did you end up doing to achieve this? – SimpleMan May 27 '16 at 06:49

0 Answers0