4

I'm currently working on my first assignment in image processing (using OpenCV in Python). My assignment is to calculate a precise score (to tenths of a point) of one to several shooting holes in an image uploaded by a user. One of the requirements is to transform the uploaded shooting target image to be from "birds-eye view" for further processing. For that I have decided that I need to find center coordinates of numbers (7 & 8) to select them as my 4 quadrilateral.

Unfortunately, there are several limitations that need to be taken into account.

Limitations:

  • resolution of the processed shooting target image can vary
  • the image can be taken in different lighting conditions
  • the image processed by this part of my algorithm will always be taken under an angle (extreme angles will be automatically rejected)
  • the image can be slightly rotated (+/- 10 degrees)
  • the shooting target can be just a part of the image
  • the image can be only of the center black part of the target, meaning the user doesn't have to take a photo of the whole shooting target (but there always has to be the center black part on it)
  • this algorithm can take a maximum of 2000ms runtime

What I have tried so far:

  1. Template matching
    • here I quickly realized that it was unusable since the numbers could be slightly rotated and a different scale
  2. Feature matching
    • I have tried all of the different feature matching types (SIFT, SURF, ORB...)
    • unfortunately, the numbers do not have that specific set of features so they matched a quite lot of false positives, but I could possibly filter them by adding shape matching, etc..
    • the biggest blocker was runtime, the runtime of only a single number feature matching took around 5000ms (even after optimizations) (on MacBook PRO 2017)
  3. Optical character recognition
    • I mostly tried using pytesseract library
    • even after thresholding the image to inverted binary (so the text of numbers 7 and 8 is black and the background white) it failed to recognize them
    • I also tried several ways of preprocessing the image and I played a lot with the tesseract config parameter but it didn't seem to help whatsoever
  4. Contour detection
    • I have easily detected all of the wanted numbers (7 & 8) as single contours but failed to filter out all of the false positives (since the image can be in different resolutions and also there are two types of targets with different sizes of the numbers I couldn't simply threshold the contour by its width, height or area)
    • After I would detect the numbers as contours I wanted to extract them as some ROI and then I would use OCR on them (but since there were so many false positives this would take a lot of time)
    • I also tried filtering them by using cv2.matchShapes function on both contours and cropped template / ROI but it seemed really unreliable

Example processed images:

1

2

3

4

5

6

As of right now, I'm lost on how to progress about this. I have tried everything I could think of. I would be immensely happy if any of you image recognition experts gave me any kind of advice or even better a usable code example to help me solve my problem.

Thank you all in advance.

JakubS
  • 133
  • 6
  • Are the images you need to process like your examples? They are quite skewed and have bad lighting- – Willem Hendriks Jul 23 '20 at 07:13
  • @user3184950 The images will usually be in better lighting / overall quality. But many users will upload images like this, therefore I have posted here both good and bad quality examples. – JakubS Jul 23 '20 at 07:24
  • 1
    How often do you want to ask the same question again?You have already asked this question twice and deleted ([How to warp perspective of whole image by its part](https://stackoverflow.com/questions/62951338/how-to-warp-perspective-of-whole-image-by-its-part)). What have you tried so far? Can you show some code? – Rabbid76 Jul 23 '20 at 14:18
  • To be honest I have not asked the same question more than once. The question you have linked was about the complete process of the perspective transformation and this one is about detecting certain digits in an image. I have removed that question specifically because I opened this one plus I felt like there wasn’t any information to gain from it for anyone else. So far I have tried everything I described in the question body. Also I could definitely provide some of my code but I feel like it wouldn’t be to any good use since there was probably problem in my approach rather than the code. – JakubS Jul 23 '20 at 14:43

1 Answers1

0

  • Find the black disk by adaptive binarization and contour (possibly blur to erase the inner features);

  • Fit an ellipse to the outline, as accurate as possible;

  • Find at least one edge of the square (Hough lines);

  • Classify the edge as one of NWSE (according to angle);

  • Use the ellipse and the line information to reconstruct the perspective transformation (it is an homography);

  • Apply the inverse homography to straighten the image and obtain the exact target center and axis;

  • Again by adaptive binarization, find the bullet holes (center/radius);

  • Rate the holes after their distance to the center, relative to the back disk radius.

If the marking scheme is variable, detect the circles (Hough circles, using the known center, or detect peaks in an oblique profile starting from the center).

If necessary, you could OCR the digits, but it seems that the score is implicitly starting at one in the outer ring.

  • Thank you for your answer. I have everything from steps 1, 2, 6, 7, and 8. But I don't really understand step 3, 4, 5. What do you mean by "finding at least one edge of the square with Hough Lines"? Do you mean one side of the rotated min area rectangle around the ellipse? What point from this line should I then select as my quadrilateral and how do I calculate its new position in the transformed image? Or did you mean something else by "line information"? I'm sorry but as I said I'm a complete beginner in image processing. Could you please provide me some basic code example? – JakubS Jul 23 '20 at 10:40
  • @JakubS: no, I mean a side of the target. Get the equation of the line. –  Jul 23 '20 at 10:42
  • Oh, unfortunately, that is not always possible :/. User can upload a photo only of the black center part of the target. Therefore sometimes there are no sides of the full shooting target. Any other idea on how to approach this? – JakubS Jul 23 '20 at 10:53
  • @JakubS: find the digits, link them to "reconstruct" two axis and find the center. –  Jul 23 '20 at 12:36
  • Well that's exactly what I'm trying to do, but I have problems finding the digits. Can you think of any approach on how to do so (other than the ones I already tried and explained in my question)? – JakubS Jul 23 '20 at 12:41
  • I already do that but there is still alot of false positives to filter out after contour detection. Thanks anyway though. – JakubS Jul 23 '20 at 12:46