cv2.matchTemplate finds wrong template in image

Question

I'm trying to create a program that knows what number is on an image with the following function:

def img_in_img(big_picture, small_picture, tamper):
    big_picture = str(big_picture)
    small_picture = str(small_picture)
    if os.path.isfile(big_picture) == False or os.path.isfile(small_picture) == False:
        return "Image does not exist"

    img = cv2.imread(big_picture,0)
    templace_loc = small_picture
    template = cv2.imread(templace_loc,0)
    w, h = template.shape[::-1]
    method = cv2.TM_CCOEFF
    
    tamper = int(tamper)

    res = cv2.matchTemplate(img,template,method)
    min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)

    top_left = max_loc
    bottom_right = (top_left[0] + w, top_left[1] + h)

    height, width, channels = cv2.imread(templace_loc).shape

    if int(top_left[0]) < int(height + tamper) and int(top_left[0]) > int(height - tamper) and int(top_left[1]) < int(width + tamper) and int(top_left[1]) > int(width - tamper):
        return True
    else:
        return False

But then when I check if 7.png is in img.png with the code

nur = "7"
if img_in_img("2020-01-14-17-36-08.537043/verification_image2.png", "verifynr/" + "old_orange" + "/" + nur + ".png", 25):
    print(Style.BRIGHT + Fore.GREEN + "Color: " + "old_orange" + ", Num: " + nur + Fore.RESET)
else:
    print(Style.BRIGHT + Fore.RED + "Color: " + "old_orange" + ", Num: " + nur + Fore.RESET)

it gives me in RED: Color: old_orange, Num: 7

but then if I check if 6.png is in img.png by changing nur from 7 to 6 it gives me in Green: Color: old_orange, Num: 6, but that's the wrong image.

I've also tried the following code:

img_rgb = cv2.imread("2020-01-14-17-36-08.537043/verification_image2.png")
img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2GRAY)
template = cv2.imread('verifynr/old_orange/7.png',0)
w, h = template.shape[::-1]

res = cv2.matchTemplate(img_gray,template,cv2.TM_SQDIFF)
threshold = 0.8
loc = np.where( res >= threshold)
pt = list(zip(*loc[::-1]))

if len(pt) >= 1:
    print("True")

which prints True, but it does that for every number png I saved.

How do I make my program recognize the 7.png in the img.png without recognizing every single number png?

img.png:

6.png:

7.png:

Template matching is not scale invariant. If the sizes do not match, it will not work well. Your "7" template is smaller than the "7" in the image. So it will not match well. You can try multi-scale template matching in this situation. Also your templates have white backgrounds. The white will match with white in your large image and give false matches. You need to create a mask for the template and use that with your template matching to avoid this issue so that it does not match white to white. You might also try using TM_CCOEFF_NORMED. — fmw42, Jan 15 '20 at 22:22
See multi-scale template matching at https://www.pyimagesearch.com/2015/01/26/multi-scale-template-matching-using-python-opencv/. Read the documentation about using a mask with your template matching. — fmw42, Jan 15 '20 at 22:28
@fmw42 So I've tried this, and it recognizes the 7 correctly (it finds it in the image on the right place), but it also finds every other number png (for example 6.png is recognized in the image, even though the 7 is in the image (when it recognizes the 6, the square arround the 7 doesnt correctly surround the 7 though). I also made the background of 7.png transparent and changed to `TM_CCOEFF_NORMED` in my own code, but then it didnt recognize both and recognized a 2.png in there. — kaci, Jan 16 '20 at 08:25
@kaci It would be helpful if you could upload at least one example of 6 and 7 in the image and their respective templates. — Rick M., Jan 16 '20 at 11:53
What did you try? Be specific. Just making the background transparent won't work. You have to make a mask and use the mask in the command. See the matchTemplate() documentation. — fmw42, Jan 16 '20 at 18:20
See my answer below for an example of how to do masked template matching (at one resolution level). — fmw42, Jan 16 '20 at 23:52

score 0 · Answer 1 · answered Jan 16 '20 at 12:24

0

Template matching not for object detection or pattern recognition, it's function is to find the most similar patch position. For detection use detectors (haar, dnn based ones, ...), for recognition use classifiers (descriptor based or nn based). '

answered Jan 16 '20 at 12:24

Andrey Smorodov

10,649
2
35
42

fmw42 · Answer 2 · 2020-01-16T23:58:31.033

Here is an example of using a template mask to do template matching at one resolution level using Python/OpenCV. The black region of the mask tells the template matching to ignore those regions when matching, so that when the template has a background color different from that in the larger image, it does not degrade the match score. Only the color regions of the template corresponding to the white in the mask contribute to the matching score. Note that masked template matching in OpenCV only works for methods TM_SQDIFF and TM_CCORR_NORMED.

Input:

Template:

Template Mask:

import cv2
import numpy as np

# read image
img = cv2.imread('logo.png')

# read template
tmplt = cv2.imread('hat.png')
hh, ww, cc = tmplt.shape

# read template mask as grayscale
tmplt_mask = cv2.imread('hat_mask.png', cv2.COLOR_BGR2GRAY)

# do template matching
corrimg = cv2.matchTemplate(img,tmplt,cv2.TM_CCORR_NORMED, mask=tmplt_mask)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(corrimg)
max_val_ncc = '{:.3f}'.format(max_val)
print("correlation match score: " + max_val_ncc)
xx = max_loc[0]
yy = max_loc[1]
print('xmatch =',xx,'ymatch =',yy)

# draw red bounding box to define match location
result = img.copy()
pt1 = (xx,yy)
pt2 = (xx+ww, yy+hh)
cv2.rectangle(result, pt1, pt2, (0,0,255), 1)

cv2.imshow('image', img)
cv2.imshow('template', tmplt)
cv2.imshow('template_mask', tmplt_mask)
cv2.imshow('result', result)
cv2.waitKey(0)
cv2.destroyAllWindows()

# save results
cv2.imwrite('logo_hat_match.png', result)

Result:

Textual Information Listed:

correlation match score: 1.000
xmatch = 417 ymatch = 44

(Apologies for borrowing and modifying the ImageMagick logo. I already had that template matching example)

Here is a Python/OpenCV example -- https://stackoverflow.com/questions/61779288/how-to-template-match-a-simple-2d-shape-in-opencv/61780200?r=SearchResults&s=1|0.0000#61780200 — fmw42, May 13 '20 at 23:18

cv2.matchTemplate finds wrong template in image

2 Answers2

Linked