OpenCV match multiple frames on image

Question

I'm a new in OpenCV and I whould like to find a solution to find multiple images with text on a some image. In the future I need that items for recognition.

First of all I've got a frame of images for template of searching. It looks like frame with transparent center. I've tried a lot of samples to match the template but they give only one result of searching: just first or second item is found, but I need all of them.

Please help me to find ways to solve the issue.

frame template:

scene:

code:

import cv2
import numpy as np

method = cv2.TM_CCOEFF_NORMED
threshold = 0.90

img_main = cv2.imread('images/garden.jpg')
template = cv2.imread('images/frame_trans.png', cv2.IMREAD_GRAYSCALE)
template_gray = template

img_main_gray = cv2.cvtColor(img_main, cv2.COLOR_BGR2GRAY)
w, h = template.shape[::-1]

res = cv2.matchTemplate(img_main_gray, template_gray, method)

cv2.normalize(res, res, 0., 1., cv2.NORM_MINMAX)
cv2.threshold(res, threshold, 1, cv2.THRESH_TOZERO)

i = 0
while i < 100:
    min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
    if method in [cv2.TM_SQDIFF, cv2.TM_SQDIFF_NORMED]:
        top_left = min_loc
    else:
        top_left = max_loc

    if max_val > threshold:
        print(top_left)
        bottom_right = (top_left[0] + w, top_left[1] + h)
        cv2.rectangle(img_main, top_left, bottom_right, (0, 0, 255), 2)
        cv2.floodFill(img_main_gray, None, top_left, 0, 0.1, 1.0)
    else:
        break
    i += 1

cv2.imwrite('sample6_output.png', img_main)
cv2.imshow('sample6', img_main)
cv2.waitKey()

Result of the script is here...

PyDev console: starting.
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32
runfile('D:/MyProjects/PyHeroRecognition/sample6.py', wdir='D:/MyProjects/PyHeroRecognition')
(71, 45)
(71, 45)
(71, 45)
(71, 45)
(71, 45)
(71, 45)
(71, 45)
(71, 45)
(71, 45)
(71, 45)

Result image:

Search Google. For example https://answers.opencv.org/question/165740/template-matching-multiple-objects/ — fmw42, Apr 10 '20 at 19:07
I've seen this example but it also didn't help me. It matches filled templates and they work as described. But I cannot run transparent frames as template. — JohanTG, Apr 11 '20 at 02:55
You can do transparency with a mask. See the matchTemplate documentation. — fmw42, Apr 11 '20 at 03:14
Yes, I guess this is a greate solution, can u show me a sample for this idea? How to create the mask? I cann't find how to work with mask in matchTemplate. Documentation has no real information just shot description of the parameter. — JohanTG, Apr 11 '20 at 11:08
The mask is just your template alpha channel. Separate the RGB image and the alpha channel. Do template matching with the RGB channel image and use the separated alpha channel as the mask. — fmw42, Apr 11 '20 at 16:59
@fmw42 Great thanks for the idea! I've crossed your code and mine but I've got just one result (see below https://stackoverflow.com/questions/61141508/opencv-match-multiple-frames-on-image/61166892#61166892). — JohanTG, Apr 12 '20 at 04:34
My code only searched for the best match. You have to write your output inside your while loop in the if condition and with different output names. — fmw42, Apr 12 '20 at 04:51

score 0 · Answer 1 · answered Apr 10 '20 at 15:50

0

For template matching algorithm whenever you find the maximum value of similarity you should set the founded pixel to zero, if you don't do this, it is like that in each iteration you find the same pixel by cv2.minMaxLoc function.

You may encounter a situation that you need to set an small area around the max pixel to zero to get best result.

I wrote a simple code for your problem, but in this context it seems that template matching is not very good

import cv2
import numpy as np

method = cv2.TM_CCORR_NORMED
threshold = 0.7


img_main = cv2.imread('images/garden.jpg')
template = cv2.imread('images/frame_trans.png', cv2.IMREAD_GRAYSCALE)
template_gray = template

img_main_gray = cv2.cvtColor(img_main, cv2.COLOR_BGR2GRAY)
w, h = template.shape[::-1]

res = cv2.matchTemplate(img_main_gray, template_gray, method)

cv2.normalize(res, res, 0., 1., cv2.NORM_MINMAX)
cv2.threshold(res, threshold, 1, cv2.THRESH_TOZERO)

i = 0

border=45  # I have added this line to control the area that must be zero
while i < 20:
    min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
    print(max_val)
    if method in [cv2.TM_SQDIFF, cv2.TM_SQDIFF_NORMED]:
        top_left = min_loc
    else:
        top_left = max_loc

    if max_val > threshold:
#         print(top_left)
        bottom_right = (top_left[0] + w, top_left[1] + h)
#         cv2.circle(img_main,max_loc,5,(0,0,255),1)
        cv2.rectangle(img_main, top_left, bottom_right, (0, 0, 255), 5)
        cv2.floodFill(img_main_gray, None, top_left, 0, 0.1, 1.0)
    i += 1
    res[max_loc[1]-border:max_loc[1]+border,max_loc[0]-border:max_loc[0]+border]=0  # set zero the max loc

img_main=cv2.cvtColor(img_main,cv2.COLOR_BGR2RGB)
import matplotlib.pyplot as plt
plt.figure(figsize=(10,12))
plt.imshow(res)
plt.figure(figsize=(10,12))
plt.imshow(img_main)

answered Apr 10 '20 at 15:50

Saeed Masoomi

1,703
1
19
33

I've tried to use floodFill on the scene as in opencv sample but it doesn't work – JohanTG Apr 10 '20 at 16:33
You are trying to find the best possible match from the `res`, and res is computed before the `while` loop so changing image without update `res` will not affect the finding rects – Saeed Masoomi Apr 11 '20 at 10:59
Yes, after applying `floodFill` nothing happens. You offer me to somehow recalculate `res` in while loop, right? I've got an idea to fill rectangle in black on the main image and then call `matchTemplate` again in the loop. – JohanTG Apr 11 '20 at 11:14
No this way maybe get you what you want, but it is time consuming and from computational view you shouldn't use that since in `match template` you are calculating all possible similarity, I don't know how much you have experienced but you can plot `res` and you can see that for each pixel you have calculated similarity already, So in other way you can interpret it as looking top-10 or top-100 in `res`, so you don't need to change image, but you have to change the `res` – Saeed Masoomi Apr 11 '20 at 11:56
For extra information please look at my answer in another question https://stackoverflow.com/questions/59485106/opencv-template-matching-for-n-best-matches-not-working/59485994#59485994 – Saeed Masoomi Apr 11 '20 at 11:57
top 10 of matching contains the same items with the same threshold. Could you help me how can I filter them in right way? – JohanTG Apr 11 '20 at 12:10
see https://stackoverflow.com/questions/61779288/how-to-template-match-a-simple-2d-shape-in-opencv/61780200?r=SearchResults&s=1|0.0000#61780200 – fmw42 May 13 '20 at 23:16

fmw42 · Answer 2 · 2020-04-11T17:17:37.030

Here is how to do template matching with a transparent template image in Python/OpenCV. Sorry about the use of Imagemagick images, but I had that example already.

Input:

Transparent Template:

import cv2
import numpy as np

# read image
img = cv2.imread('logo.png')

# read template with alpha
tmplt = cv2.imread('hat_alpha.png', cv2.IMREAD_UNCHANGED)
hh, ww = tmplt.shape[:2]

# extract template mask as grayscale from alpha channel and make 3 channels
tmplt_mask = tmplt[:,:,3]
tmplt_mask = cv2.merge([tmplt_mask,tmplt_mask,tmplt_mask])

# extract templt2 without alpha channel from tmplt
tmplt2 = tmplt[:,:,0:3]

# do template matching
corrimg = cv2.matchTemplate(img,tmplt2,cv2.TM_CCORR_NORMED, mask=tmplt_mask)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(corrimg)
max_val_ncc = '{:.3f}'.format(max_val)
print("correlation match score: " + max_val_ncc)
xx = max_loc[0]
yy = max_loc[1]
print('xmatch =',xx,'ymatch =',yy)

# draw red bounding box to define match location
result = img.copy()
pt1 = (xx,yy)
pt2 = (xx+ww, yy+hh)
cv2.rectangle(result, pt1, pt2, (0,0,255), 1)

cv2.imshow('image', img)
cv2.imshow('template2', tmplt2)
cv2.imshow('template_mask', tmplt_mask)
cv2.imshow('result', result)
cv2.waitKey(0)
cv2.destroyAllWindows()

# save results
cv2.imwrite('logo_hat_match2.png', result)

Result showing bounding box of match location:

score 0 · Answer 3 · answered Apr 12 '20 at 04:32

@fmw42 Great thanks for the idea! I've crossed your code and mine but I've got just one result (see below). My next question: how to capture few results?

import cv2
import numpy as np

method = cv2.TM_CCORR_NORMED
threshold = 0.90

# read scene image
main_img = cv2.imread('images/garden.jpg')

# read template with alpha
template = cv2.imread('images/frame_trans.png', cv2.IMREAD_UNCHANGED)
h, w = template.shape[:2]

# extract template mask as grayscale from alpha channel and make 3 channels
tmplt_mask = template[:, :, 3]
tmplt_mask = cv2.merge([tmplt_mask, tmplt_mask, tmplt_mask])

# extract templt2 without alpha channel from tmplt
tmplt2 = template[:, :, 0:3]

res = cv2.matchTemplate(main_img, tmplt2, method, mask=tmplt_mask)

output_img = main_img.copy()

i = 0
while i < 100:
    min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
    if method in [cv2.TM_SQDIFF, cv2.TM_SQDIFF_NORMED]:
        top_left = min_loc
    else:
        top_left = max_loc

    if max_val > threshold:
        print("%s: %s" % (max_val, top_left))
        bottom_right = (top_left[0] + w, top_left[1] + h)
        cv2.rectangle(output_img, top_left, bottom_right, (0, 0, 255), 2)
        cv2.floodFill(res, None, top_left, 0, 0.1, 1.0)
    else:
        break
    i += 1

cv2.imwrite('sample8_output.png', output_img)
cv2.imshow('sample8', output_img)
cv2.waitKey()

Console output:

PyDev console: starting.
Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32
runfile('D:/MyProjects/PyHeroRecognition/sample8_alpha.py', wdir='D:/MyProjects/PyHeroRecognition')
0.9918215870857239: (71, 45)
>>>

Result image output

My code only searched for the best match. You have to write your output inside your while loop in the if condition and with different output names. Put a counter in your condition in the while loop. Then use something like `cv2.imwrite("result_{0}.png".format(i), output_img)` — fmw42, Apr 12 '20 at 04:55

OpenCV match multiple frames on image

3 Answers3