Using opencv matchtemplate for blister pack inspection

Question

I am doing a project in which I have to inspect pharmaceutical blister pack for missing tablets.

I am trying to use opencv's matchTemplate function. Let me show the code and then some results.

int match(string filename, string templatename)
{
    Mat ref = cv::imread(filename + ".jpg");
    Mat tpl = cv::imread(templatename + ".jpg");
    if (ref.empty() || tpl.empty())
    {
        cout << "Error reading file(s)!" << endl;
        return -1;
    }

    imshow("file", ref);
    imshow("template", tpl);

    Mat res_32f(ref.rows - tpl.rows + 1, ref.cols - tpl.cols + 1, CV_32FC1);
    matchTemplate(ref, tpl, res_32f, CV_TM_CCOEFF_NORMED);

    Mat res;
    res_32f.convertTo(res, CV_8U, 255.0);
    imshow("result", res);

    int size = ((tpl.cols + tpl.rows) / 4) * 2 + 1; //force size to be odd
    adaptiveThreshold(res, res, 255, ADAPTIVE_THRESH_MEAN_C, THRESH_BINARY, size, -128);
    imshow("result_thresh", res);

    while (true) 
    {
        double minval, maxval, threshold = 0.8;
        Point minloc, maxloc;
        minMaxLoc(res, &minval, &maxval, &minloc, &maxloc);

        if (maxval >= threshold)
        {
            rectangle(ref, maxloc, Point(maxloc.x + tpl.cols, maxloc.y + tpl.rows), CV_RGB(0,255,0), 2);
            floodFill(res, maxloc, 0); //mark drawn blob
        }
        else
            break;
    }

    imshow("final", ref);
    waitKey(0);

    return 0;
}

And here are some pictures.

The "sample" image of a good blister pack:

The template cropped from "sample" image:

Result with "sample" image:

Missing tablet from this pack is detected:

But here are the problems:

I currently don't have any idea why this happens. Any suggestion and/or help is appreciated.

The original code that I followed and modified is here: http://opencv-code.com/quick-tips/how-to-handle-template-matching-with-multiple-occurences/

Answer 1

I found a solution for my own question. I just need to apply Canny edge detector on both image and template before throwing them to matchTemplate function. The full working code:

int match(string filename, string templatename)
{
    Mat ref = cv::imread(filename + ".jpg");
    Mat tpl = cv::imread(templatename + ".jpg");
    if(ref.empty() || tpl.empty())
    {
        cout << "Error reading file(s)!" << endl;
        return -1;
    }

    Mat gref, gtpl;
    cvtColor(ref, gref, CV_BGR2GRAY);
    cvtColor(tpl, gtpl, CV_BGR2GRAY);

    const int low_canny = 110;
    Canny(gref, gref, low_canny, low_canny*3);
    Canny(gtpl, gtpl, low_canny, low_canny*3);

    imshow("file", gref);
    imshow("template", gtpl);

    Mat res_32f(ref.rows - tpl.rows + 1, ref.cols - tpl.cols + 1, CV_32FC1);
    matchTemplate(gref, gtpl, res_32f, CV_TM_CCOEFF_NORMED);

    Mat res;
    res_32f.convertTo(res, CV_8U, 255.0);
    imshow("result", res);

    int size = ((tpl.cols + tpl.rows) / 4) * 2 + 1; //force size to be odd
    adaptiveThreshold(res, res, 255, ADAPTIVE_THRESH_MEAN_C, THRESH_BINARY, size, -64);
    imshow("result_thresh", res);

    while(1) 
    {
        double minval, maxval;
        Point minloc, maxloc;
        minMaxLoc(res, &minval, &maxval, &minloc, &maxloc);

        if(maxval > 0)
        {
            rectangle(ref, maxloc, Point(maxloc.x + tpl.cols, maxloc.y + tpl.rows), Scalar(0,255,0), 2);
            floodFill(res, maxloc, 0); //mark drawn blob
        }
        else
            break;
    }

    imshow("final", ref);
    waitKey(0);

    return 0;
}

Any suggestion for improvement is appreciated. I am strongly concerned about performance and robustness of my code, so I am looking for all ideas.

There are 2 things that got my nerves now: the lower Canny threshold and the negative constant on adaptiveThreshold function.

Edit: Here is the result, as you asked :)

Template:

Test image, missing 2 tablets:

Canny results of template and test image:

matchTemplate result (converted to CV_8U):

After adaptiveThreshold:

Final result:

Answer 2

I don't think think the adaptive threshold is a good choice.

What you need to do here is called non-maximum suppression. You have an image with multiple local maxima, and you want to remove all pixels that are not local maxima.

cv::dilate(res_32f, res_dilated, null, 5);
cv::compare(res_32f, res_dilated, mask_local_maxima, cv::CMP_GE);
cv::set(res_32f, 0, mask_local_maxima)

Now all pixels in the res_32f image that are not local maxima are set to zero. All the maximum pixels are still at their original value, so you can adjust the threshold later in the line

double minval, maxval, threshold = 0.8;

All local maxima should also now be surrounded by enough zeroes that the floodfill will not extend too far.

Now I think you should be able to adjust the threshold to exclude all false positives.

---

If this is not enough, here is another suggestion:

Instead of just one template, I would run the search with multiple templates; your current template,and one with a tablet from the right side and the left side of the pack. Due to perspective these tablets look quite a bit different. Keep track of the found tablets so you do not detect the smae tablet multiple times.

With these multiple templates you can raise the threshold even higher.

---

One further refinement: if the detection is still too erratic, try blurring your template and search image with a Gaussian blur. This will remove fine details and noise that may throw of the matchTemplate function, while leaving the larger structures intact.

Using a canny filter instead seems unreliable to me: It seems to rely on the fact that a removed tablet region will have more edges at the center. But I am not sure if this will always be the case; and you discard a lot of information about color and brightness with the canny filter, so I would expect worse results.

(that said, if it works for you, it works)

Answer 3

Have you tried the Surf algorithm in order to get more detailed descriptors? You could try to collect descriptor for both the full and the empty sample image. And perform different action for each one of thr object detected.