(-215:Assertion failed) corr.rows <= img.rows + templ.rows - 1 && corr.cols <= img.cols + templ.cols - 1 in function 'cv::crossCorr' (matchTemplate)

Question

I've run into issues with win32gui when trying to grab a real-time video stream of an application. I've seen I can use ImageGrab from PIL and based on this video Computer Screen Recording using Python & OpenCV I think I can use it instead of win32gui

I'm trying to learn python by writing a bot, the below code should grab images from a specified folder, load them into an array, converts them into a format OpenCV can use and then attempts to find any or all of them on my application window haystack

I can't find any details on google of the error I'm getting:

C:\Users\coyle\OneDrive\froggy-pirate-master\avoidShips>C:/Users/coyle/AppData/Local/Programs/Python/Python39/python.exe c:/Users/coyle/OneDrive/froggy-pirate-master/avoidShips/avoidships4.py
Traceback (most recent call last):
  File "c:\Users\coyle\OneDrive\froggy-pirate-master\avoidShips\avoidships4.py", line 41, in <module>
    loadImages()
  File "c:\Users\coyle\OneDrive\froggy-pirate-master\avoidShips\avoidships4.py", line 22, in loadImages
    return matchTemplate(image_list)
  File "c:\Users\coyle\OneDrive\froggy-pirate-master\avoidShips\avoidships4.py", line 32, in matchTemplate
    result = cv.matchTemplate(haystack, needle_img, cv.TM_CCOEFF_NORMED)
cv2.error: OpenCV(4.5.1) C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-wvn_it83\opencv\modules\imgproc\src\templmatch.cpp:588: error: (-215:Assertion failed) corr.rows <= img.rows + templ.rows - 1 && corr.cols <= img.cols + templ.cols - 1 in function 'cv::crossCorr'

And my code:

def loadImages():
    # Intialise empty array
    image_list = []
    # Get list of all images in directory
    directory = glob.glob(r"C:\Users\*.png")

    # Add images to image_list
    for img in directory:
        ship_img = cv.imread(img, 0)
        image_list.append(ship_img)
    return matchTemplate(image_list)

def matchTemplate(image_list):
    # Video Loop
    while True:
        haystack_img = ImageGrab.grab()
        haystack_img_np = np.array(haystack_img)
        haystack = cv.cvtColor(haystack_img_np, cv.COLOR_BGR2GRAY)
        
        # Object Detection
        for ships in image_list:
            needle_img = cv.imread(str(image_list), cv.IMREAD_UNCHANGED)
            result = cv.matchTemplate(haystack, needle_img, cv.TM_CCOEFF_NORMED)
            cv.imshow('Result', haystack)

            if cv.waitKey(1) == 27:
                break

        cv.destroyAllWindows()

loadImages()
matchTemplate()

As a test, I've tried doing the same thing using static images and it works so I'm not sure where I'm going wrong.

import cv2 as cv
import glob

# load source images
directory = glob.glob(r'C:\Users\*.jpg')
# empty list to store the source images
image_list = []

for img in directory:
    ships_img = cv.imread(img, 0)
    image_list.append(ships_img)

haystack_img = cv.imread(r'C:\Users\both.jpg')
haystack_img = cv.cvtColor(haystack_img, cv.COLOR_BGR2GRAY)

#loop for matching
for ships in image_list:
    
    #save the dimensions of the needle images
    (H, W) = ships.shape[:2]
    result = cv.matchTemplate(haystack_img, ships, cv.TM_CCOEFF)
    min_val, max_val, min_loc, max_loc = cv.minMaxLoc(result)
    top_left = max_loc
    
    bottom_right = (top_left[0] + W, top_left[1] + H)
    cv.rectangle(haystack_img, top_left, bottom_right, 255, 2)

cv.imshow('Result', haystack_img)
cv.waitKey(0)

Answer 1

I can't test it but you simply try to load image which you already have in memory

You have

needle_img = cv.imread(str(image_list), cv.IMREAD_UNCHANGED)

but image_list has already loaded image, not filenames.
Besides imread() needs sinlge filename but you try use it with some list converted to string.

You should use directly

needle_img = ships

---

I think it should be

def loadImages():
    # Intialise empty array
    image_list = []
    
    # Get list of all images in directory
    directory = glob.glob(r"C:\Users\*.png")

    # Add images to image_list
    for img in directory:
        ship_img = cv.imread(img, 0)  # <-- here you load all images
        image_list.append(ship_img)
        
    return image_list   # I preferr to send back data instead of running `matchTemplate`


def matchTemplate(image_list):
    # Video Loop
    while True:
        haystack_img = ImageGrab.grab()
        haystack_img_np = np.array(haystack_img)
        haystack = cv.cvtColor(haystack_img_np, cv.COLOR_BGR2GRAY)
        
        # Object Detection
        for ships in image_list:
            # you don't have to load images because you already have them in `image_list`
            #needle_img = cv.imread(str(image_list), cv.IMREAD_UNCHANGED)
            
            needle_img = ships
            
            result = cv.matchTemplate(haystack, needle_img, cv.TM_CCOEFF_NORMED)
            cv.imshow('Result', haystack)

            if cv.waitKey(1) == 27:
                break

        cv.destroyAllWindows()

# --- main ---

image_list = loadImages()
matchTemplate(image_list)

---

BTW:

In normal open(), read() you get error if it has problem to open or read file but in OpenCV imread() does't raise error when it can't load image but it gives None - but you don't check if you get None - and you don't know that there was problem to load it - and later when you try to use this value in next command (matchTemplate) then it shows error. But real problem was with imread()