3

I am using SIFT for feature detection and calcOpticalFlowPyrLK for feature tracking in images. I am working on low resolution images (590x375 after cropping) taken from Microsoft kinect. 

// feature detection
cv::Ptr<Feature2D> detector = cv::xfeatures2d::SIFT::create();
detector->detect(img_1,keypoints_1);
KeyPoint::convert(keypoints_1, points1, vector<int>());

// feature tracking
vector<float> err;
Size winSize=Size(21,21);
TermCriteria termcrit=TermCriteria(TermCriteria::COUNT+TermCriteria::EPS, 30, 0.01);
calcOpticalFlowPyrLK(img_1, img_2, points1, points2, status, err, winSize, 1, termcrit, 0, 0.001);

I ran this on consective images of steady scene (just to get idea) taken from same camera position at rate of 30fps. To eyes, images looks same but somehow the calcOpticalFlowPyrLK in not able to track same features from one image to another. Position (x,y coordinates) should be same in detected feature and tracked feature. Somehow it isn't.

As per AldurDisciple suggestion, I think I am detecting noise as features. The black images below are difference between consuctive elements, shows noise. Next ones are original images and then images with detected features.

My goal is to use information to find change in robot's position over time.

I used

GaussianBlur( currImageDepth, currImageDepth, Size(9,9), 0, 0);

for noise but it didn't help.

Find complete code in here enter image description here

enter image description here

enter image description here

enter image description here

enter image description here

enter image description here

Ruturaj
  • 630
  • 1
  • 6
  • 20
  • 1
    Print the distances (L2 norm) between the input and output points and see if they make sense. If the motions are small, you can reduce the search window size and check. Also plot the FAST keypoints in your image and see if they are good enough for tracking. – dhanushka Nov 02 '16 at 08:46
  • I drew lines between detected and tracked coordinates. It makes sense when I move camera, but when I don't I still can see few lines, which are giving me wrong result. I tried drawing circle around detected points, they did not make much sense to me. – Ruturaj Nov 03 '16 at 18:54
  • 1
    When there's no motion, what fraction of the points show a motion, and what about their magnitude? May be you can use this info to discard these motions. Anyway, getting totally rid of false alarms may not be possible. FAST was designed to run in real-time and may not be very accurate. Use a detector such as SIFT or SURF and compare the quality of the keypoints. You may be able to adjust the FAST parameters based on this comparison, if you must use FAST. – dhanushka Nov 04 '16 at 00:29
  • i am not sure what fraction, magnitude was significant, but Lucas kanade tracked it as motion in optical flow. I tried goodfeaturestotrack beside FAST and adjusting FAST parameters which didn't work very well, I will give SIFT and SURF a try tomorrow. Also I am using RAW image from kinect, should I do any processing on it? – Ruturaj Nov 04 '16 at 04:47

1 Answers1

3

I think there are two factors you should take into account:

  1. Your scene basically consists of 3 homogeneous regions, hence the FAST points in these regions will likely be generated by noise in the image. Since the noise pattern can be completely different in two successive images, the best match for some of the points may be at a completely different position in the image.

  2. Your image already has quite a low resolution and the 3 in the parameter list of the calcOpticalFlowPyrLK function mean that you require the function to track the points using a pyramid of 4 levels. This means that the points will first be tracked in the image resized by a factor 2^3 = 16 (i.e. ~ 36x23 image), then in the image resized by a factor 2^2 = 8 (i.e. ~ 73x46 image) and so on. An initial resolution of 36x23 is much too low for an image with almost no texture.

To solve your problem, you can try to use only two pyramid levels (i.e. pass 1 instead of 3) or even a single level (i.e. pass 0 instead of 3). But keep in mind that the noise issue implies that in general you will always have a few false matches.

On the other hand, tracking points in a static scene with no camera motion very much seems like an artificial problem. In real scenarios, you will probably be more interested in tracking motion in the scene or a static scene using a moving camera, in which case using several pyramid levels will be useful.

BConic
  • 8,750
  • 2
  • 29
  • 55
  • I am trying smoothing on images now, the robot moves at many places and the image was one of the examples. My main objective is to track motion, but I was getting non zero values for steady images, which made me think that my result with motion will have error as well. The robot moves very slow and captures images at 30FPS. I process images and then get next one, I don't process all if not possible. So how many pyramid levels should I use? Also I replaced FAST with SIFT and trying out SURF now. I have access to depth images as well, will it work if go for Depth image instead of current RGB? – Ruturaj Nov 05 '16 at 23:59
  • I drew circle around detected features, they are mostly in corners, or where 2 planes meet, very few (2-4) were in homogeneous regions. I subtracted consecutive images and I got some noise (non zero values) where planes meet or darker areas. I will add images tomorrow morning. – Ruturaj Nov 06 '16 at 06:03