opencv c++ compare keypoint locations in different images

Question

When comparing 2 images via feature extraction, how do you compare keypoint distances so to disregard those that are obviously incorrect?

I've found when comparing similar images against each other, most of the time it can fairly accurate, but other times it can throws matches that are completely separate.

So I'm after a way of looking at the 2 sets of keypoints from both images and determining whether the matched keypoints are relatively in the same locations on both. As in it knows that keypoints 1, 2, and 3 are so far apart on image 1, so the corresponding keypoints matched on image 2 should be of a fairly similar distance away from each other again.

I've used RANSAC and minimum distance checks in the past but only to some effect, they don't seem to be as thorough as I'm after.

(Using ORB and BruteForce)

EDIT

Changed "x, y, and z" to "1, 2, and 3"

EDIT 2 -- I'll try to explain further with quick Paint made examples:

Say I have this as my image:

And I give it this image to compare against:

Its a cropped and squashed version of the original, but obviously similar.

Now, say you ran it through feature detection and it came back with these results for the keypoints for the two images:

The keypoints on both images are in roughly the same areas, and proportionately the same distance away from each other. Take the keypoint I've circled, lets call it "Image 1 Keypoint 1".

We can see that there are 5 keypoints around it. Its these distances between them and "Image 1 Keypoint 1" that I want to obtain so to compare them against "Image 2 Keypoint 1" and its 5 surround keypoints in the same area (see below) so as to not just compare a keypoint to another keypoint, but to compare "known shapes" based off of the locations of the keypoints.

--

Does that make sense?

Answer 1

Keypoint matching is a problem with several dimensions. These dimensions are:

• spatial distance, ie, the (x,y) distance as measured from the locations of two keypoints in different images
• feature distance, that is, a distance that describes how much two keypoints look alike.

Depending on your context, you do not want to compute the same distance, or you want to combine both. Here are some use cases:

• optical flow, as implemented by opencv's sparse Lucas-Kanade optical flow. In this case, keypoints called good features are computed in each frame, then matched on a spatial distance basis. This works because the image is supposed to change relatively slowly (the input frames have a video framerate);
• image stitching, as you can implement from opencv's features2d (free or non-free). In this case, the images change radically since you move your camera around. Then, your goal becomes to find stable points, ie, points that are present in two or more images whatever their location is. In this case you will use feature distance. This also holds when you have a template image of an object that you want to find in query images.

In order to compute feature distance, you need to compute a coded version of their appearance. This operation is performed by the DescriptorExtractor class. Then, you can compute distances between the output of the descriptions: if the distance between two descriptions is small then the original keypoints are very likely to correspond to the same scene point.

Pay attention when you compute distances to use the correct distance function: ORB, FREAK, BRISK rely on Hamming distance, while SIFt and SURF use a more usual L2 distance.

Match filtering

When you have individual matches, you may want to perform match filtering in order to reject good individual matches that may arise from scene ambiguities. Think for example of a keypoint that originates from the corner of a window of a house. Then it is very likely to match with another window in another house, but this may not be the good house or the good window.

You have several ways of doing it:

• RANSAC performs a consistency check of the computed matches with the current solution estimate. Basically, it picks up some matches at random, computes a solution to the problem (usually a geometric transform between 2 images) and then counts how many of the matchings agree with this estimate. The estimate with the higher count of inliers wins;
• David Lowe performed another kind of filtering in the original SIFT paper. He kept the two best candidates for a match with a given query keypoint, ie, points that had the lowest distance (or highest similarity). Then, he computed the ratio similarity(query, best)/similarity(query, 2nd best). If this ratio is too low then the second best is also a good candidate for a match, and the matching result is dubbed ambiguous and rejected.

Exactly how you should do it in your case is thus very likely to depend on your exact application.

Your specific case

In your case, you want to develop an alternate feature descriptor that is based on neighbouring keypoints. The sky is obviously the limit here, but here are some steps that I would follow:

1. make your descriptor rotation and scale invariant by computing the PCA of the keypoints :

   // Form a matrix from KP locations in current image
   cv::Mat allKeyPointsMatrix = gatherAllKeypoints(keypoints); 
   
   // Compute PCA basis
   cv::PCA currentPCA(allKeyPointsMatrix, 2);
   
   // Reproject keypoints in new basis
   cv::Mat normalizedKeyPoints = currentPCA.project(allKeyPointsMatrix);
   

2. (optional) sort the keypoints in a quadtree or kd-tree for faster spatial indexing

3. Compute for each keypoint a descriptor that is (for example) the offsets in normalized coordinates of the 4 or 5 closest keypoints
4. Do the same in your query image
5. Match keypoints from both mages based on these new descriptors.

Answer 2

What is it you are trying to do exactly? More information is needed to give you a good answer. Otherwise it'll have to be very broad and most likely not useful to your needs.

And with your statement "determining whether the matched keypoints are relatively in the same locations on both" do you mean literally on the same x,y positions between 2 images?

I would try out the SURF algorithm. It works extremely well for what you described above (though I found it to be a bit slow unless you use gpu acceleration, 5fps vs 34fps).

Here is the tutorial for surf, I personally found it very useful, but the executables are for linux users only. However you can simply remove the OS specific bindings in the source code and keep only the opencv related bindings and have it compile + run on linux just the same.

https://code.google.com/p/find-object/#Tutorials

Hope this helped!

Answer 3

You can do a filter on the pixels distance between two keypoint. Let's say matches is your vector of matches, kp_1 your vector of keypoints on the first picture and kp_2 on the second. You can use the code above to eliminate obviously incorrect matches. You just need to fix a threshold.

double threshold= YourValue;
vector<DMatch> good_matches;
for (int i = 0; i < matches.size(); i++)
{
    double dist_p = sqrt(pow(abs(kp_1[matches[i][0].queryIdx].pt.x - kp_2[matches[i][0].trainIdx].pt.x), 2) + pow(abs(kp_1[matches[i][0].queryIdx].pt.y - kp_2[matches[i][0].trainIdx].pt.y), 2));
    if (dist_p < threshold)
    {
        good_matches.push_back(matches[i][0]);
    }
}