-1

I'm running Sikulix v2.0.5 on a linux machine (ubuntu 22.04 LTS). In order to ingnore background I try to use template images with transparent areas. But as soon as I use transparency Sikulix behave strage in finding the correct matches on screen. with a normal (70% percent) comparison it finds 'match' that have nothing to do with the template. Only a 99% percent comparison finds the correct areas, but as so the match must fit 99% percent. So slightly different pictures compared to the non transparent part of the template can't be found.

What I tried:

If I have i.e. a template image of a folder on my desktop and I use it with Sikulix in a normal way by just using its filename "Image.png" (70% comparison) sikulix finds only that exact area where I took the template from. So it reacts as expected like the screenshot of dialog 'pattern properties' shows the correct area in red.

When I edit the template image in order to make the background transparent I gets the following template image with transparency. If I use this image as template, sikulix finds areas that have nothing to do with the template image as this screenshot of the dialog 'pattern properties' shows.

Only if one uses a 99% comparison the right areas are found as this screenshot shows.

But as one can see, only folders which look exactly like the templates are found. The first one in the upper left corner ist not found due to its slightly different picture.

So what I would expect is, that if I use i.e. a 70% comparison in conjunction with transparency, sikulix should compare that 70% of the NON TRANSPARENT part of the template should fit for a match, thus ignoring a background but find similar pictures.

Is Sikulix the right destination for placing this question or should I switch over to OpenCV to ask this question.

Regards

I made two more templates differing in only one pixel. The first one with lower right pixel black and the second with lower right pixel transparent. The first one leads with the 70% accuracy to exactly one region on the desktop. The second template leads to many regions that are recognized erroneously.

akazen
  • 1
  • 3
  • if you don't use opencv, the opencv tag would be inappropriate. if this "sikuli" thing uses opencv, still it's a "sikuli" question. I'm also not seeing any source code, so this question might not even be appropriate for Stack Overflow. perhaps https://superuser.com/ would be a better fit? if you're prepared to program this, perhaps look into pyautogui. – Christoph Rackwitz May 18 '23 at 18:39
  • Thanks for your hint, but the Sikulix WEB-Site https://sikulix.github.io/docs/help/questions/ tells me to place questions here. So I have done. – akazen May 18 '23 at 22:14
  • So my question is. Does opencv always behave like this or does only Sikuli-x a wrong handling of what opencv does correct. – akazen May 18 '23 at 22:38
  • blame sikuli and most people's (sikuli's programmers') insufficient grasp of how `matchTemplate` works, when to use it, what parameters to use, etc. in principle, your situation is not challenging to solve with `matchTemplate`. I'm only here because of the opencv tag. I have no interest in sikuli. – Christoph Rackwitz May 19 '23 at 09:52
  • Yes sorry, I just ran into this situation by using sikulix. So I'm new to this materia of template recognition. But as I wrote my question here, I want to get a little bit behind-the-scenes. As I can unterstand for now, the more details (sharp transitions) a template has, the better it can be distinguished - right? So as the main recognition depends on opencv I've to understand opencv?! So why do you remove the tag opencv. Just because you are annoyed of sikulix users?! – akazen May 20 '23 at 13:49
  • Or do you mean that this poor distinguish is due to an incorrect implementation in sikulix? – akazen May 20 '23 at 14:19

1 Answers1

0

The non-transparent area of the image does not contain many details. So given a low similarity, areas on the screen are found, that are just coloured similar. So it makes sense, that only with a high similarity the expected matches are found.

The base image (no transparency) has enough details, so even with the standard similarity of 70% only expected matches are found.

Generally it is recommended to raise the standard similarity to a higher value (> 0.8, using Settings.MinSimilarity = ...;). This on one hand forces you to make better screenshots (little background, concentrate on the key visuals of the image) and on the other hand leads to more reliable matches. A wanted similarity of 0.95 and above normally leads to only exact matches.

This is RaiMan from SikuliX

Raimund Hocke
  • 1
  • As I can unterstand for now, the more details (sharp transitions) a template has, the better it can be distinguished - right? So large areas of **same exact colour** are less information than much **structured** ones. So how should I make screenshot to be independant of a backgroud colour of a i.e. a button but have a good template for distingquishing. As I understand at least the Edge of the button should be part of the template. But there is already the mixture of the background with the foreground to to antialiasing. – akazen May 20 '23 at 14:01
  • Above 10 - 20 % of background pixels (of all image pixels) around the center pixel area are usually ok. The higher the value the higher the risk for false positives. Antialiasing zones do not matter (not many extra pixels). So if you discard the background with transparency, then also 10 - 20 % should be ok. But it is rather easy to make a short test sequence to get a feeling yourself. RaiMan / SikuliX – Raimund Hocke May 21 '23 at 15:24
  • In investigated a little bit more on the item. And it look like Sikulix has definitly a problem with transparancy itself. I made two templates that differ in only **ONE Pixel**. The first one has the lower right pixel **black**. The second one has the lower left pixel **transparent**. The rest of the template images is fully equal. The first one is recognized clear. The second one is recognized nearly everwhere on screen. Look at my edit of the question. – akazen May 24 '23 at 01:38