0

I am trying to write a python script that searching for an image on the screen using cv2.matchTemplate fucntion, and then extracting the x and y coordinates of that image and move the mouse to that location. But for some reason when I am running the code the mouse moves to the end of the screen and I do not understand why. Can someone help me figer it out?

this is my code:

import numpy as np
import mss
import cv2
import time
import pyautogui

# the templates to search for by order
TEMPLATES_BY_ORDER = ['EmailField.png',
                      'PasswordField.png', 'CheckBox.png', 'SignInButton.png']


def main():
    time.sleep(5)
    currentImage = 0

    with mss.mss() as sct:
        # Get the size of the primary monitor
        monitor = {"top": 0, "left": 0, "width": pyautogui.size()[
            0], "height": pyautogui.size()[1]}

        while currentImage < 1:
            # the path for the current template image
            template_img_path = f"images/{TEMPLATES_BY_ORDER[currentImage]}"

            # reading both the screen recording frame and the template image to search for
            screen_frame = np.array(sct.grab(monitor))[:, :, :3]
            template_img = cv2.imread(template_img_path, cv2.IMREAD_COLOR)

            screen_frame_dementions = screen_frame.shape

            resized_img = cv2.resize(
                template_img, (screen_frame_dementions[1], screen_frame_dementions[0]))

            # setting the next image in line
            currentImage += 1

            # matching the template to the screen record
            result = cv2.matchTemplate(
                screen_frame, resized_img, cv2.TM_CCOEFF_NORMED)

            # getting the coordinates of the template in the recording
            __, __, __, max_loc = cv2.minMaxLoc(result)
            x, y = max_loc
            pyautogui.moveTo(x, y)
            if cv2.waitKey(25) & 0xFF == ord("q"):
                cv2.destroyAllWindows()
                break


if __name__ == "__main__":
    main()

This is the screen that I am letting the code record and search in. enter image description here And this is the test.png image that I am trying to search for enter image description here

bob
  • 73
  • 7
  • 1
    Two issues. First, I do not recommend using TM_CCOEFF_NORMED. Use TM_SQDIFF_NORMED or TM_CCORR_NORMED. Second , your template image is not the same size (scale) as the region in your image that you want to find. The scales are different. cv2.matchTemplate() requires the same scale unless test at various scales. Also note that cv2.matchTemplate will work on 3 channel color images. So you do not have to convert to grayscale – fmw42 Jan 13 '23 at 21:26
  • what do I need to change in my code to make them the same scale? – bob Jan 13 '23 at 21:28
  • You need to measure the dimensions of the region in the image and then note the size of the template. Then use cv2.resize() to make the template the same dimensions as the region in the image. If this is a one time thing and all the screen images have the same region (size), then just measure it manually. Otherwise, if the regions are different in different images, then the text will likely be different and your template will need to be different. – fmw42 Jan 13 '23 at 21:31
  • You can also just crop your image around the region you want and use that as the template. It will then be guaranteed to be the same size. Then it can be used for other similar images. – fmw42 Jan 13 '23 at 21:34
  • related: https://stackoverflow.com/questions/75111029/i-need-to-automate-the-filling-of-a-html-form-in-a-web-browser-how – Christoph Rackwitz Jan 13 '23 at 22:16
  • just use pyautogui to find the picture, then use pyautogui to move the mouse pointer. none of this requires explicit use of OpenCV. -- if you insist on using OpenCV, you should take a look at a (normalized version of) the matchTemplate result. it might give some insight into what each matching mode produces. – Christoph Rackwitz Jan 13 '23 at 22:17
  • I tried using pyautogui as well but it doing the same thing – bob Jan 13 '23 at 22:22
  • both your template and the haystack (in the current version of the question at least) have matching scales. I don't see an issue there. in case that _is_ an issue after all, make sure to directly crop the template from a screenshot. no resizing/resampling, no taking from another screenshot. -- it's very likely the matching mode being unsuitable. also, with SQDIFF, you're looking for a minimum. – Christoph Rackwitz Jan 13 '23 at 22:23
  • I updated my code to make sure that the images are in the same scale using the ```.shape```. But its still don't work for me – bob Jan 13 '23 at 22:26
  • scale means the same object in the picture is the same size in pixels in both pictures (your pictures satisfy this already). don't look at `.shape` for this aspect. – Christoph Rackwitz Jan 14 '23 at 14:56

0 Answers0