I'm trying to find and identify agent/character icons in an image for a project (specifically the agent icons that are outlined with a green border). I only began learning OpenCV a month ago, and the most success I've had was using template matching to find the agent icons.
But the greatest problem I'm facing is how to separate bad matches vs good matches besides using a threshold. Sometime when I try template matching an agent icon, my program will point to the wrong part of the image. For example, when I'm template matching Omen_Icon image into MiniMap1, the program will point to the wrong location on MiniMap1FindingOmen as indicated by the red dot. Similarly, when I'm trying to template match overlapping images, because an agent icon is hidden behind another icon, unsurprisingly, my program will point to the wrong part of the image. For example, in MiniMap2, we can see that the the Raze_Icon image is behind another agent icon and my program will point to the wrong location in MiniMap2FindingRaze. I was hoping that I could use a threshold value to separate bad matches vs good matches, but my bad and good matches frequently have similar values from template matching with Sum of Squared Differences, making it very hard to find a definitive line to separate bad and good matches.
My Template Matching Code:
import numpy as np
import cv2
from pathlib import Path
from isolatingGreenAgentIcons import refinedColorThresholds
# Compares an image to a template to see if a match is found
mapName = "Haven3"
listOfRoundNumbers = [1]
listOfAgentsToFind = ["sova"]
# Get the path to the parent directory
parentParentDirectory = Path.cwd().parent.parent
miniMapPath = parentParentDirectory.joinpath("miniMaps")
# print("MiniMap Path: ", miniMapPath)
def agentFindingInMiniMap(agent, imgRGB):
baseTemplate = cv2.imread(str(parentParentDirectory / f"agentIconImgs/{agent}.png"), 0)
# Loaded in the template
h, w = baseTemplate.shape
imgGray = cv2.cvtColor(imgRGB, cv2.COLOR_BGR2GRAY)
# Created a copy of the Colored Image but in Gray scale
result = cv2.matchTemplate(imgGray, baseTemplate, cv2.TM_SQDIFF_NORMED)
# SQDIFF_NORMED is the Sum of Squared Differences, which means similar images have a smaller difference
# aka the values are btw 0 and 1, and the closer the min value of result is to 0, the more closely the
# base template and image patch match
maxDiff = 0.5
# Right now, I have a high threshold because I can't find a value that will consistently separate bad matches and good matches
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(result)
if min_val < maxDiff:
print(f"We found a(n) {agent} with our Base Template! Min value: ", min_val)
top_left_loc = min_loc # min_location is in (column, row) format, which is why you add w first before h
bottom_right_loc = (top_left_loc[0] + w, top_left_loc[1] + h)
xCoord = int((top_left_loc[0] + bottom_right_loc[0])/2)
yCoord = int((top_left_loc[1] + bottom_right_loc[1])/2)
imgRGB = cv2.circle(imgRGB, (xCoord,yCoord), radius=3, color=(0, 0, 255), thickness=-1)
# Dotted where the agent is found on the minimap
cv2.imshow("Match", imgRGB)
cv2.waitKey(0)
cv2.destroyAllWindows()
else:
print(f"No match was found => Agent {agent} was not found ", min_val)
if __name__ == "__main__":
for roundNumber in listOfRoundNumbers:
try:
print("Round Number: ", roundNumber)
imgMiniMap = refinedColorThresholds(mapName, roundNumber)
for agent in listOfAgentsToFind:
agentFindingInMiniMap(agent, imgMiniMap)
except Exception as e:
print(f"An error occurred while processing round {roundNumber}: {str(e)}")
I've also tried using feature detecting and matching by using SIFT or ORB and FLANN based Matcher or Brute-Force Matcher and the results were inaccurate (I'm guessing because there are not enough pixels in my templates to find features).
Edit:
Just to clarify, the reason why MiniMap1 and MiniMap1FindingOmen look different besides the red dot was because I used a few masks to isolate the agent icons. This way, I thought there would be a lower possibility of a bad match because the areas that didn't have the agent icons were blacked out. The same goes for MiniMap2 and MiniMap2FindingRaze.
