Why is OpenCV GPU CUDA template matching so much slower than CPU?

Question

I have compiled the newest available OpenCV 4.5.4 version for use with the newest CUDA 11.5 with fast math enabled running on a Windows 10 machine with a GeForce RTX 2070 Super graphics card (7.5 arch). I'm using Python 3.8.5.

Runtime results:

• CPU outperforms GPU (matching a 70x70 needle image in a 300x300 source image)
• biggest GPU bottleneck is the need to upload the files to the GPU before template matching
• CPU takes around 0.005 seconds while the GPU takes around 0.42 seconds
• Both methods end up finding a 100% match

Images used:

Source image

Needle image

Python code using CPU:

import cv2
import time

start_time = time.time()
src = cv2.imread("cat.png", cv2.IMREAD_GRAYSCALE)
needle = cv2.imread("needle.png", 0)

result = cv2.matchTemplate(src, needle, cv2.TM_CCOEFF_NORMED)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(result)
print("CPU --- %s seconds ---" % (time.time() - start_time))

Python code using GPU:

import cv2
import time

start_time = time.time()
src = cv2.imread("cat.png", cv2.IMREAD_GRAYSCALE)
needle = cv2.imread("needle.png", 0)

gsrc = cv2.cuda_GpuMat()
gtmpl = cv2.cuda_GpuMat()
gresult = cv2.cuda_GpuMat()

upload_time = time.time()
gsrc.upload(src)
gtmpl.upload(needle)
print("GPU Upload time --- %s seconds ---" % (time.time() - upload_time))

match_time = time.time()
matcher = cv2.cuda.createTemplateMatching(cv2.CV_8UC1, cv2.TM_CCOEFF_NORMED)
gresult = matcher.match(gsrc, gtmpl)
print("GPU Match time --- %s seconds ---" % (time.time() - match_time))

result_time = time.time()
resultg = gresult.download()
min_valg, max_valg, min_locg, max_locg = cv2.minMaxLoc(resultg)
print("GPU Result time --- %s seconds ---" % (time.time() - result_time))
print("GPU --- %s seconds ---" % (time.time() - start_time))

Even if I wouldn't take the time it takes to upload the files to the GPU into consideration the matching time alone takes more than 10x of the whole process on the CPU. My CUDA is installed correctly, I have run other tests where the GPU outperformed the CPU by a lot, but the results for template matching are really disappointing so far.

Why is the GPU performing so badly?

Answer 1

In answer to your question:

1. You said that other tasks were better suited to the GPU. I read the Python CUDA documentation. It suggests that you are correct. Some tasks are better suited to the CPU and some are better suited to the CPU. Without getting into registries and stuff I would have to learn to tell you, I can say that what you write makes sense in reference to the documentation.
2. I don't see the actual times here. Also, it seems that this bottleneck is to be expected: the CPU is on the motherboard, soldered on with a better connection to the memory. The GPU is a card, attached with an extended plug that has limitations that a motherboard doesn't. Also, it is not really a troublesome bottleneck because it is not congested. 2.1 Someone else wrote that my hypothesis about the computer's design is "nonsense." I beg to differ. This tiny dataset could point out the difference in speed between the card and the board. The data originates on the card, then goes to the board's registries. That tiny set could easily make the difference noticeable.
3. What I have read about architecture and the CUDA documentation suggests that your results are not abnormal. The CUDA modules might be better used with large datasets. The advantage provided is that the GPU and CPU can work simultaneously, not in competition.