wrong contours and wrong output of handwritten digit recognition AI model

Question

After training a model for handwritten digit recognition, when I provide the input image it shows the wrong contours and the wrong output. The input data contains 5 digits, but the output say 10-15 digits. Even it does not create right triangle. Following is the code to train the model and the code to give new image input

import cv2
import numpy as np
from keras.datasets import mnist
from keras.layers import Dense, Flatten
from keras.layers.convolutional import Conv2D
from keras.models import Sequential
from keras.utils import to_categorical
 import matplotlib.pyplot as plt

(X_train, y_train), (X_test, y_test) = mnist.load_data()

 print ("Shape of X_train: {}".format(X_train.shape))
 print ("Shape of y_train: {}".format(y_train.shape))
 print ("Shape of X_test: {}".format(X_test.shape))
 print ("Shape of y_test: {}".format(y_test.shape))

 X_train = X_train.reshape(60000, 28, 28, 1)
 X_test = X_test.reshape(10000, 28, 28, 1)

y_train = to_categorical(y_train)
y_test = to_categorical(y_test)

## Declare the model
model = Sequential()

## Declare the layers
layer_1 = Conv2D(32, kernel_size=3, activation='relu', input_shape=(28, 28, 1))
layer_2 = Conv2D(64, kernel_size=3, activation='relu')
layer_3 = Flatten()
layer_4 = Dense(10, activation='softmax')

## Add the layers to the model
model.add(layer_1)
model.add(layer_2)
model.add(layer_3)
model.add(layer_4)

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=3)

model.save('Digit_Recognition_Model_2.model')

and to give new image input and make predictions

import tensorflow as tf
import numpy as np
import cv2
import matplotlib.pyplot as plt

model=tf.keras.models.load_model('Digit_Recognition_Model_2.model')

image = cv2.imread('test_images/test3.jpeg')
grey = cv2.cvtColor(image.copy(), cv2.COLOR_BGR2GRAY)
ret, thresh = cv2.threshold(grey.copy(), 75, 255, cv2.THRESH_BINARY_INV)
contours, _ = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL, 
cv2.CHAIN_APPROX_SIMPLE)
preprocessed_digits = []
for c in contours:
    x,y,w,h = cv2.boundingRect(c)

    # Creating a rectangle around the digit in the original image (for displaying the digits fetched via contours)
cv2.rectangle(image, (x,y), (x+w, y+h), color=(0, 255, 0), thickness=2)

# Cropping out the digit from the image corresponding to the current contours in the for loop
digit = thresh[y:y+h, x:x+w]

# Resizing that digit to (18, 18)
resized_digit = cv2.resize(digit, (18,18))

# Padding the digit with 5 pixels of black color (zeros) in each side to finally produce the image of (28, 28)
padded_digit = np.pad(resized_digit, ((5,5),(5,5)), "constant", constant_values=0)

# Adding the preprocessed digit to the list of preprocessed digits
preprocessed_digits.append(padded_digit)
print("\n\n\n----------------Contoured Image--------------------")
plt.imshow(image, cmap="gray")
plt.show()

inp = np.array(preprocessed_digits)

for digit in preprocessed_digits:
    prediction = model.predict(digit.reshape(1, 28, 28, 1))  
    print(prediction.argmax())

now given an image with digits 504192 it outputs 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 8 2 5 4 5 0 the output after threshhold and drawing contours is !https://drive.google.com/file/d/1-H5Ov3SKyuCCkUUsSuq9gXqyn99J2c7T/view?usp=sharing

may be the problem is with image processing. what's the output image after threshold and after drawing contours?? — lnx, Jun 07 '20 at 15:27
@Inx I know the problem is with image processing as I am new to this. I have attached a drive link to the output after the threshold and after drawing contours. — Arjun Bajaj, Jun 07 '20 at 16:16
what's the output after thresholding?? Also provide original image so that i also can try — lnx, Jun 07 '20 at 21:46
try cv2.CHAIN_APPROX_NONE instead of cv2.CHAIN_APPROX_SIMPLE — lnx, Jun 07 '20 at 21:52
why don't you take a look at every crop from image then decide how to preprocess to avoid false contours — lnx, Jun 07 '20 at 22:00
@Inx I tried cv2.CHAIN_APPROX_NONE but the problem still persist. — Arjun Bajaj, Jun 08 '20 at 06:46

score 0 · Answer 1 · answered Jun 08 '20 at 10:25

0

Do few changes

orig=cv2.imread('numbers.png')
img=cv2.imread('numbers.png',0)

ret, thresh = cv2.threshold(img.copy(), 150, 255, cv2.THRESH_BINARY_INV)
contours, _ = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
digits=[]
for c in contours:
   x,y,w,h = cv2.boundingRect(c)
   cv2.rectangle(orig, (x,y), (x+w, y+h), color=(0, 255, 0), thickness=2)
   digit = thresh[y:y+h, x:x+w]
   padded_digit = np.pad(digit, ((10,10),(10,10)), "constant", constant_values=0)
   digit=cv2.resize(padded_digit,(28,28))
   digits.append(digit)
inp = np.array(digits).reshape((len(digits),28,28,1))
y_pred = model.predict(inp)  #instead of one by one predict, all at once

These are segmented digits

if you want you can use little dilation for better results

answered Jun 08 '20 at 10:25

lnx

318
2
12

I tried doing this but it messed up the contours and now it is not predicting even a single digit. – Arjun Bajaj Jun 09 '20 at 07:51
Divide the problem into 2 . First is segmentation digits (I have provided the code) . Second is recognition If the digits still not getting recognized then problem may be, we have segmented the digits from a image which is relatively higher than 28x28 that's why, while resizing the digits some vital information may be lost, so I think you should try dilation on original image and then segmentation, this may help. Let me know if this works or not – lnx Jun 09 '20 at 18:24
@Inx I tried giving it a 28x28 image so that we dont have to resize and it gave the correct answer. I think grayscaling, thresholding is correctly done. The only problem is to convert the image to (1,28,28,1). – Arjun Bajaj Jun 11 '20 at 05:09

wrong contours and wrong output of handwritten digit recognition AI model

1 Answers1