Why doesn't RANSAC remove all the outliers in SIFT matches?

Question

I use SIFT to detect, describe feature points in two images as follows.

void FeaturePointMatching::SIFTFeatureMatchers(cv::Mat imgs[2], std::vector<cv::Point2f> fp[2])
{
    cv::SiftFeatureDetector  dec;
    std::vector<cv::KeyPoint>kp1, kp2;

    dec.detect(imgs[0], kp1);
    dec.detect(imgs[1], kp2);

    cv::SiftDescriptorExtractor ext;
    cv::Mat desp1, desp2;

    ext.compute(imgs[0], kp1, desp1);
    ext.compute(imgs[1], kp2, desp2);

    cv::BruteForceMatcher<cv::L2<float> > matcher;
    std::vector<cv::DMatch> matches;
    matcher.match(desp1, desp2, matches);

    std::vector<cv::DMatch>::iterator iter;

    fp[0].clear();
    fp[1].clear();
    for (iter = matches.begin(); iter != matches.end(); ++iter)
    {
        //if (iter->distance > 1000)
        //  continue;
        fp[0].push_back(kp1.at(iter->queryIdx).pt);
        fp[1].push_back(kp2.at(iter->trainIdx).pt);
    }

    // remove outliers
    std::vector<uchar> mask;
    cv::findFundamentalMat(fp[0], fp[1], cv::FM_RANSAC, 3, 1, mask);

    std::vector<cv::Point2f> fp_refined[2];
    for (size_t i = 0; i < mask.size(); ++i)
    {
        if (mask[i] != 0)
        {
            fp_refined[0].push_back(fp[0][i]);
            fp_refined[1].push_back(fp[1][i]);
        }
    }

    std::swap(fp_refined[0], fp[0]);
    std::swap(fp_refined[1], fp[1]);
}

In the above code, I use findFundamentalMat() to remove outliers, but in the result img1 and img2 there are still some bad matches. In the images, each green line connects the matched feature point pair. And please ignore red marks. I can not find anything wrong, could anyone give me some hints? Thanks in advance.

score 1 · Accepted Answer · answered Mar 16 '16 at 07:22

1

RANSAC is just one of the robust estimators. In principle, one can use a variety of them but RANSAC has been shown to work quite well as long as your input data is not dominated by outliers. You can check other variants on RANSAC like MSAC, MLESAC, MAPSAC etc. which have some other interesting properties as well. You may find this CVPR presentation interesting (http://www.imgfsr.com/CVPR2011/Tutorial6/RANSAC_CVPR2011.pdf)

Depending on the quality of the input data, you can estimate the optimal number of RANSAC iterations as described here (https://en.wikipedia.org/wiki/RANSAC#Parameters)

Again, it is one of the robust estimator methods. You may take other statistical approaches like modelling your data with heavy tail distributions, trimmed least squares etc.

In your code you are missing the RANSAC step. RANSAC has basically 2 steps:

generate hypothesis (do a random selection of data points necessary to fit your mode: training data).
model evaluation (evaluate your model on the rest of the points: testing data)
iterate and choose the model that gives the lowest testing error.

answered Mar 16 '16 at 07:22

Luca

10,458
24
107
234

Thanks for your thorough explanation. Doesn't `cv::findFundamentalMat(fp[0], fp[1], cv::FM_RANSAC, 3, 1, mask);` do a complete RANSAC step as you said? – Anonymous Mar 16 '16 at 07:50
Ah yes. Looking at the opencv docs, it seems it does. I have a feeling though that these parameters may not be optimised. So the first thing I would try is to reduce Param1 from 3 to lower values and see what I get. I would also set Param2 to something like 0.99 as it may reduce the number of iterations and computation time. A certainty of 1 seems a bit too much. – Luca Mar 16 '16 at 09:44
Thanks. I will have a try. – Anonymous Mar 17 '16 at 02:11
Also note that the way this opencv method is implemented seems very monolithic and offers little control. If changing these two parameters do not work, try breaking it up to get a bit more flexibility on the RANSAC parameters and stopping conditions. – Luca Mar 17 '16 at 07:47

score 0 · Answer 2 · answered Mar 16 '16 at 06:52

0

RANSAC stands for RANdom SAmple Consensus, it does not remove outliers, it selects a group of points to calculate the fundamental matrix for that group of points. Then you need to do a re projection using the fundamental matrix just calculated with RANSAC to remove the outliers.

answered Mar 16 '16 at 06:52

Luis

127
1
11

Thanks for your help. It seems time-costly to do RANSAC and reprojection. Is there any other better method? Thanks again! – Anonymous Mar 16 '16 at 06:57
Well, it may seem so, but it isn't. Reprojecting a few coordinates is relatively cheap compared to extract features and is being done hundreds of times during a findFundamentalMat call – Gilfoyle Mar 16 '16 at 09:20
@Gilfoy I see. Thanks. – Anonymous Mar 17 '16 at 02:08

score 0 · Answer 3 · answered Mar 16 '16 at 11:23

Like any algorithm, ransac is not perfect. You can try to run other disponible (in the opencv implementation) robust algorithms, like LMEDS. And you can reiterate, using the last points marked as inliers as input to a new estimation. And you can vary the threshold and confidence level. I suggest run ransac 1 ~ 3 times and then run LMEDS, that does not need a threshold, but only work well with at least +50% of inliers.

And, you can have geometrical order problems:

*If the baseline between the two stereo is too small the fundamental matrix estimation can be unreliable, and may be better to use findHomography() instead, for your purpose.

*if your images have some barrel/pincushin distortion, they are not in conformity with epipolar geometry and the fundamental matrix are not the correct mathematical model to link matches. In this case, you may have to calibrate your camera and then run undistort() and then process the output images.

Why doesn't RANSAC remove all the outliers in SIFT matches?

3 Answers3