How to extract digit in image that overide rect area

Question

Now, I have a project that use image processing about detect digit but I don't know how to extract when digit override the rect area.

Here's the input:

I want to get something like the bottom pictures:

Answer 1

The digits can be extracted using OpenCV using the concept of contours. The following implementation is in Python.

Go through the following steps:

• Convert your image to gray-scale
• Apply appropriate threshold so that the digits are in white while everything else is in black.
• Now find contours. Since you have small stars they will also get picked up. To avoid picking them up pass a suitable area above which contours must be found.
• Compute the convex hull for each contour and mask them on the original image.

You need to work with individual convex hull after extracting the contours.

Code:

img = cv2.imread(r'C:\Users\Jackson\Desktop\2_new.jpg', 1)
img2 = img.copy()
cv2.imshow("original.jpg", img)

gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

_, threshed_img = cv2.threshold(gray, 127, 255, cv2.THRESH_BINARY)
image, contours, hier = cv2.findContours(threshed_img, cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_NONE)

#--- Black image to be used to draw individual convex hull ---
black = np.zeros_like(img)
#cv2.imshow("black.jpg", black)

contours = sorted(contours, key=lambda ctr: cv2.boundingRect(ctr)[0]) #added by OP : this sorts contours left to right, so images come in order

for cnt in contours:
    if cv2.contourArea(cnt) > 200:
        hull = cv2.convexHull(cnt)

        black2 = black.copy()

        #--- Here is where I am filling the contour after finding the convex hull ---
        cv2.drawContours(black2, [hull], -1, (255, 255, 255), -1)
        g2 = cv2.cvtColor(black2, cv2.COLOR_BGR2GRAY)
        _, t2 = cv2.threshold(g2, 127, 255, cv2.THRESH_BINARY)
        cv2.imshow("t2.jpg", t2)

        masked = cv2.bitwise_and(img2, img2, mask = t2)    
        cv2.imshow("masked.jpg", masked)

        print(len(hull))
        cv2.waitKey(0)

cv2.destroyAllWindows()

You should be able to get something like the following: