How to adjust the threshold for template matching in openCV (java)?

Question

I am running template matching using openCV 3.4.7 Android SDK (java). The code work almost perfectly; when the template is match, it draws a rectangle on the matching area. The problem is that even when there is no match, it draws a random rectangle. I think that happens because the threshold is not set correctly. If so, can someone please help me out?

Here's the code:

public static void run(String inFile, String templateFile, String outFile,
                    int match_method) {
        Mat img = Imgcodecs.imread(inFile);
        Mat templ = Imgcodecs.imread(templateFile);

        // / Create the result matrix
        int result_cols = img.cols() - templ.cols() + 1;
        int result_rows = img.rows() - templ.rows() + 1;
        Mat result = new Mat(result_rows, result_cols, CvType.CV_32FC1);

        // / Do the Matching and Normalize
        Imgproc.matchTemplate(img, templ, result, match_method);
        Core.normalize(result, result, 0, 1, Core.NORM_MINMAX, -1, new Mat());

        // / Localizing the best match with minMaxLoc
        Core.MinMaxLocResult mmr = Core.minMaxLoc(result);

        Point matchLoc;
        if (match_method == Imgproc.TM_SQDIFF
                || match_method == Imgproc.TM_SQDIFF_NORMED) {
            matchLoc = mmr.minLoc;
        } else {
            matchLoc = mmr.maxLoc;
        }

        // / Show me what you got
        Imgproc.rectangle(img, matchLoc, new Point(matchLoc.x + templ.cols(),
                matchLoc.y + templ.rows()), new Scalar(0, 0, 128));

        // Save the visualized detection.
        System.out.println("Writing " + outFile);
        Imgcodecs.imwrite(outFile, img);
}

Answer 1

You can use Imgproc.TM_CCOEFF_NORMED or Imgproc.TM_CCORR_NORMED and mmr.maxVal >= 0.8. It should take care of most of your false positives.

Sample Code:

import org.opencv.core.*;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.imgproc.Imgproc;

import java.io.File;
import java.nio.file.Files;

public class templateMatchingTester {

    private static String str = null;

    static {
        if (str == null) {
            str = "initialised";
            nu.pattern.OpenCV.loadShared();
            System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
        }

    }

    private static Mat createMatrixFromImage(String imagePath) {
        Mat imageMatrix = Imgcodecs.imread(imagePath);
        Mat greyImage = new Mat();
        Imgproc.cvtColor(imageMatrix, greyImage, Imgproc.COLOR_BGR2GRAY);
        return greyImage;
    }

    private static boolean matchTemplate(String pathToInputImage,String pathToTemplate){

        Mat inputImage = createMatrixFromImage(pathToInputImage);
        Mat templateImage = createMatrixFromImage(pathToTemplate);

        // Create the result matrix
        int result_cols = inputImage.cols() - templateImage.cols() + 1;
        int result_rows = inputImage.rows() - templateImage.rows() + 1;
        Mat result = new Mat(result_rows, result_cols, CvType.CV_8UC1);
        int match_method;
        match_method = Imgproc.TM_CCOEFF_NORMED;//Imgproc.TM_CCORR_NORMED;
        Imgproc.matchTemplate(inputImage, templateImage, result, match_method);
        Core.MinMaxLocResult mmr = Core.minMaxLoc(result);
        double minMatchQuality = 0.85; 
        System.out.println(mmr.maxVal);
        if (mmr.maxVal >= minMatchQuality){
            return true;
        } else
        return false;
    }

    public static void main(String args[]) {

        String template = "path/to/your/templateImage";
        final File folder = new File("path/to/your/testImagesFolder/");
        int matchCount = 0;
        for (final File fileEntry : folder.listFiles()){
            if (matchTemplate(fileEntry.getPath(),template)){
                matchCount+=1;
            }else
                System.out.println(fileEntry.getPath());
        }
        System.out.println(matchCount);

    }
}

Answer 2

Use a normed match method to ensure your match value is [0..1].

Replace this line

Core.normalize(result, result, 0, 1, Core.NORM_MINMAX, -1, new Mat());

with a thresholding operation. Otherwise a best match of 0.9 would become 1 by the second normalization and you would lose the actual match "quality" information.

Normalizing the result of the template matching will always result in your best match being 1 making it impossible to discard a bad match.

Answer 3

i wrote an app that would take a screenshot of the game overwatch and attempt to tell who is on each team. using template matching and open cv. project need to iterate over the result image and check values.

OpenCVUtils.getPointsFromMatAboveThreshold(result, 0.90f)

public static void scaleAndCheckAll(String guid){       
    Mat source = imread(IMG_PROC_PATH + guid);  //load the source image
    Mat scaledSrc = new Mat(defaultScreenshotSize, source.type());
    resize(source, scaledSrc, defaultScreenshotSize);
    Mat sourceGrey = new Mat(scaledSrc.size(), CV_8UC1);
    cvtColor(scaledSrc, sourceGrey, COLOR_BGR2GRAY);        

    for (String hero : getCharacters()) {
        Mat template = OpenCVUtils.matFromJar(TEMPLATES_FOLDER + hero + ".png", 0); //load a template
        Size size = new Size(sourceGrey.cols()-template.cols()+1, sourceGrey.rows()-template.rows()+1);
        Mat result = new Mat(size, CV_32FC1);
        matchTemplate(sourceGrey, template, result, TM_CCORR_NORMED);// get results
        Scalar color =  OpenCVUtils.randColor();
        List<Point> points = OpenCVUtils.getPointsFromMatAboveThreshold(result, 
0.90f);
        for (Point point : points) {
            //rectangle(scaledSrc, new Rect(point.x(),point.y(),template.cols(),template.rows()), color, -2, 0, 0);
            putText(scaledSrc, hero, point, FONT_HERSHEY_PLAIN, 2, color);
        }
    }
    String withExt = IMG_PROC_PATH + guid +".png";
    imwrite(withExt,  scaledSrc);
    File noExt = new File(IMG_PROC_PATH + guid);
    File ext = new File(withExt);
    noExt.delete();
    ext.renameTo(noExt);                
}

the other method.

public static List<Point> getPointsFromMatAboveThreshold(Mat m, float t){
    List<Point> matches = new ArrayList<Point>();
    FloatIndexer indexer = m.createIndexer();
    for (int y = 0; y < m.rows(); y++) {
        for (int x = 0; x < m.cols(); x++) {
            if (indexer.get(y,x)>t) {
                System.out.println("(" + x + "," + y +") = "+ indexer.get(y,x));
                matches.add(new Point(x, y));                   
            }
        }           
    }       
    return matches;
}

you can just get the first from the list or see how close they are if you expect multiple matches.