1

I am new to the image recognition stuff, so please be gentle :-). At the moment I'm looking to match a list of png files to screenshots: [Right click any image + open in new tab to see that bigger version]

  • Screenshots (image shape: 370x370x3):

  • Match against: (There are 4 different ones with each coming in 2 variants; a normal and a shiny one) (image shape: 256x256x4)

From the metadata, I already could boil down the possible matches (fyi: the files start with "pokemon_icon_585*").

What I've been trying to figure out is how to 'see' which one it is. (ie: which has the highest likeliness). What I've already been up to:

  1. the 'center' of the image is x / 2, y * (2/3). So that also makes it easier to detect more relevant points. I've been trying to figure out how to do a check on color with a weight based on the distance from that center. Not much luck there -- likely because of my limited knowledge.
  2. feature detection via OpenCV (ORB_create) + matching (FBMatcher) (the found matches don't match up properly -- meaning that for example an ear is matched with a foot, but the top matches themselves are all located on the animal, so that might be usable):
  1. I've been trying to figure out how to apply a mask (taken by the png and applied via transformation onto the screengrab). This would make me able to remove the background so that will not interfere with the detection.
  2. And next step would be to compare overal features/colors.

So in the above list I'm stuck trying to find information on 3. What can I search for? What libraries/methods should I use? And what for step 4? How would you do this? As you can see from the screengrabs and the 'templates' to match against, they're fairly similar to a human. But they are turned around a bit. As for the matches: screengrab 1 & 2 should be matching template 7, and screenshot 3 & 4 should match template 5.

Or of course if my approach is wrong, please do tell me how you would approach it with some keywords! That'd be nice!

The current way that I'm going is to create a bitmask to remove the background:

# Read it
img1 = cv2.imread('kkgbgwwpwu04ji.png')
# Find the edges
edges = cv2.Canny(img1, 100, 200)

# Remove the horizontal line at the bottom:
horizontal_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (25,1))
detected_lines = cv2.morphologyEx(edges.copy(), cv2.MORPH_OPEN, horizontal_kernel, iterations=2)
cnts, _ = cv2.findContours(detected_lines, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
img2 = edges.copy()
for c in cnts:
    cv2.drawContours(img2, [c], -1, (0,0,0), 2)

final = np.vstack([edges[:x1], img2[x1:x2], edges[x2:]])

# Get a bitmask
kernel = np.ones((5, 5), np.uint8)
mask = cv2.morphologyEx(edges.copy(), cv2.MORPH_CLOSE, kernel, iterations=10)

But how to remove the arc on top? I know I could just cut off the picture, but some pokemons are very large, and as such would be cut off as well.

This is final:

And this is mask:

Steven Van Ingelgem
  • 872
  • 2
  • 9
  • 25
  • See matchTemplate() assuming the images are the same size. It permits the use of a mask to avoid processing where the mask is black. – fmw42 Dec 20 '20 at 06:17
  • As far as I understood, matchTemplate means the template should be an exact match. Which is not in this case. – Steven Van Ingelgem Dec 20 '20 at 07:27
  • post more pictures then. we need to get a feel for your situation. I hope you aren't dealing with thumbnail size data. matchTemplate doesn't need an exact match but it doesn't deal well with deformable objects, or viewpoint changes... those pokemon probably move around on the screen. – Christoph Rackwitz Dec 21 '20 at 00:24
  • Thanks for your comment @ChristophRackwitz. I updated my post with more images and information. Hopefully this helps you help me ;-) – Steven Van Ingelgem Dec 21 '20 at 07:21
  • ah good. the model appears to be static and only varies in color. I'd try getting the gradient (laplacian), convert that to grayscale, then use those representations for comparisons. matchTemplate should work to find the location of the template, or if you already know precisely where the template will be, use simple sum of absolute differences or similar method. – Christoph Rackwitz Dec 21 '20 at 20:32
  • there's another question about a similar task here: https://stackoverflow.com/questions/65381698/opencv-match-template-only-on-a-specified-grid-of-positions – Christoph Rackwitz Dec 21 '20 at 20:33
  • as for calculating a mask... the simplest would be to get a picture of just the background, and then, in a given image with an object in it, label those pixels as background that match the background template exactly. that's a variant of "greenscreen". in the case of matchTemplate, it respects the template's mask for scoring, so you don't have to have a mask for the "haystack" you're looking at. – Christoph Rackwitz Dec 21 '20 at 20:36
  • @ChristophRackwitz: the "exact match" is not possible because they apply 2 different things: 1 is a scaled "type background picture", which is available. But the other is a dynamic effect (for example for water it are a few bubbles that float randomly across the screen). For this example, it are some solar glares/flares that open and close at random. (I updated my question -- do you think this way is worth pursuing?) – Steven Van Ingelgem Jan 03 '21 at 05:36

0 Answers0