How can I improve the quality of the recognition of playing cards?

Question

I am trying to build a classification of playing cards. The goal is that I use an image of a playing card as input and the script then outputs whether it is a playing card and if so, which one.

For this I have written the following comparison method, which I call with the input image and all comparison images, i.e.:

Input image vs. 2_C
Input image vs. 2_H
Input image vs. 2_D
[...]

Compare method:

def compare_images(img1, img2):
    percent = -1
    detector = cv2.SIFT_create()
   
    kp1, des1 = detector.detectAndCompute(img1,None)
    kp2, des2 = detector.detectAndCompute(img2,None)

    bf = cv2.BFMatcher_create()
    matches = bf.knnMatch(des1,des2,k=2)


    # Apply ratio test
    good = []
    for m,n in matches:
        if m.distance < 0.75*n.distance:
            good.append([m])
            a=len(good)
            percent=(a*100)/len(kp2)

    return percent

It then outputs which playing card matches how well with the input image, but not always reliably..., example:

Input image: 2_C, Output of cv2.drawMatchesKnn(img2, kp2, img1, kp1, matches, None):

Output of percent value:

Match of <2_C> & <2_J>: 197.2972972972973
Match of <2_C> & <2_C>: 28.115942028985508
Match of <2_C> & <A_C>: 27.0
Match of <2_C> & <3_H>: 26.344086021505376
Match of <2_C> & <2_S>: 25.28735632183908
Match of <2_C> & <2_D>: 23.529411764705884
Match of <2_C> & <7_H>: 20.95808383233533
Match of <2_C> & <2_H>: 18.807339449541285
Match of <2_C> & <5_H>: 16.80327868852459
Match of <2_C> & <4_H>: 15.104166666666666
Match of <2_C> & <3_C>: 14.031180400890868
Match of <2_C> & <3_S>: 12.546125461254613
[...]

How can I improve the quality of detection?

matches alone aren't enough. you need to `findHomography` (next tutorial in opencv docs) or a lesser transformation. and then you would need to consider if the homography is plausible (corners of the card mapped anywhere near sensibly, or mostly rotation and uniform scaling, no shear and other stuff) — Christoph Rackwitz, Aug 14 '22 at 11:41
For this task comparison with features not ideal solution. I recommend just try `absDiff` on images. If it work bad, look on the bad cases. Probably `erosion/dilation` operations can help with result images. Also resize image can give some speedup and quality improvements. — Gralex, Aug 15 '22 at 11:41
One more approach is threshold image, find blobs and check that it has same properties of blobs in same locations. https://learnopencv.com/blob-detection-using-opencv-python-c/ + add additional color check. — Gralex, Aug 15 '22 at 11:44
Please try asking on [ai.se]. This is not a discussion forum, and we are interested specifically at the code level, rather than general design of an algorithm. — Karl Knechtel, Sep 08 '22 at 00:47
your approach can't be fixed. discard it. local feature descriptors are too generic and too low level in terms of receptive field. you need to actually recognize the shapes and numbers/letters on these cards. then you have useful features. — Christoph Rackwitz, Sep 08 '22 at 10:32

score 0 · Answer 1 · answered Sep 08 '22 at 00:41

It it hard to make reliable recognition with this approach as it may have limited accuracy.

But you could play with feature descriptor and matcher parameters. For example instead of ratio test you can set fixed threshold and count number of good matches:

7ofclubs vs 7ofclubs = 187 good matches:

7ofclubs vs 8ofclubs = 85 good matches:

7ofclubs vs 7ofhearts = 13 good matches:

7ofhearts vs 8ofclubs = 0 good matches:

Complete example:

import cv2

fe = cv2.AKAZE_create(cv2.AKAZE_DESCRIPTOR_KAZE_UPRIGHT, 0, 3, 0.007)

image1 = cv2.imread("card1.png")
image2 = cv2.imread("card2.jpg")

sf = 700. / image1.shape[1]
image1 = cv2.resize(image1, (0, 0), fx=sf, fy=sf)
sf = 700. / image2.shape[1]
image2 = cv2.resize(image2, (0, 0), fx=sf, fy=sf)

kp1, desc1 = fe.detectAndCompute(image1, None)
kp2, desc2 = fe.detectAndCompute(image2, None)

knn_matcher = cv2.DescriptorMatcher_create(cv2.DESCRIPTOR_MATCHER_BRUTEFORCE)
matches = knn_matcher.knnMatch(desc1, desc2, k=2)
good_matches = []

for neighbors in matches:
    for m in neighbors:
        if m.distance < 0.2:
            good_matches.append(m)

print(f"Number of good matches: {len(good_matches)}")

out = cv2.drawMatches(image1, kp1, image2, kp2, good_matches, None)
cv2.imwrite("matches.jpg", out)

Additionally we can use the fact that a card consist of a fixed number of elements: suit (clubs, hearts, etc) and a number on a white background. We can segment these elements:

and locate keypoints that belongs to specific segment and count this keypoint only one per semgent to make sure that the number is matched and the suit is matched.

But from my perspective a more reliable method is to train a neural network.

How can I improve the quality of the recognition of playing cards?

1 Answers1