Connecting model to bounding boxes

Question

First of all, thank you for taking the time to read my question.

I'm trying to predict text in bounding boxes. Somehow it looks like there is no connection between the boxes and the model.

I used EAST text detection to create bounding boxes and trained a CNN model with Keras. Now i'm retrieving input through my webcam with opencv. readNet the EAST model and load_model my trained CNN.

I trained my model with black text words on a white background.

When I'm trying to connect the two it creates the right boxes in the video capture, however it doesn't provide the right prediction class.

This is what the last part of my code looks like:

   # loop over the bounding boxes
    for (startX, startY, endX, endY) in boxes:
        # scale the bounding box coordinates based on the respective
        # ratios
        startX = int(startX * rW)
        startY = int(startY * rH)
        endX = int(endX * rW)
        endY = int(endY * rH)
    
        orig_res =cv2.resize(orig,(200,200))
        orig_res = orig_res.reshape(1,200,200,3)
        
        prediction = model.predict(orig_res)
        max_prediction = np.argmax(prediction[0])
        my_prediction = myList[max_prediction]
            
        cv2.putText(orig,str(my_prediction), (startX, startY),
            cv2.FONT_HERSHEY_SIMPLEX, 1.2, (0, 0, 255), 3)
        
        # draw the bounding box on the image
        cv2.rectangle(orig, (startX, startY), (endX, endY), (0, 255, 0), 2)
        
    # show the output image
    cv2.imshow("Detect_text", orig)

myList stands for the folder names of the different words. These are the names that the model should predict. As you can see in this screenshot, it doesn't predict "100%" ..

This is a selection of what the dataset looks like:

Can someone provide a solution for this? Many thanks!

are you sure the labels you used to train your model and folder names are aligned ? I mean are you sure that the label 0 of your model corresponds to the first item in `myList`. It may help debugging if you can provide the code on how you created `myList`, and the code on how you created your image data generator — Alka, Mar 28 '21 at 14:10
Hi Alka, this is the code for creating the myList myList = os.listdir(folder) — roet, Mar 28 '21 at 14:21
After this I used the folder path in the generator by class_mode='categorical' — roet, Mar 28 '21 at 14:23
Yes first I used ImageDataGenerator and then I created a train and validation set with flow_from_directory — roet, Mar 28 '21 at 14:35

score 0 · Answer 1 · answered Mar 28 '21 at 14:39

0

It looks like a mismatch between the ordering of the labels you used during training and the list of labels you created at inference from the list of folder names. The reason is that os.listdir returns filenames in an arbitrary order, whereas keras image generator will sort the folder names before mapping them to class indices.

You can ensure to have the same ordering by creating list of labels as follow:

# suppose your training data generator is named train_gen
myList = [item[0] for item in sorted(list(train_gen.class_indices.items()), key=lambda x: x[1])]

This way you've your labels names used during training and during inference aligned

answered Mar 28 '21 at 14:39

Alka

767
1
6
13

When you say it isn't working what is exactly is the problem, is it that it still doesn't predict the class "100%" for your example image ? – Alka Mar 28 '21 at 14:59
Yes. It doesn't matter how and were I show the 100% image, it only shows other label names. – roet Mar 28 '21 at 15:09
well. Maybe your model is not well trained. Did you try with other images of other classes to see if the predictions are consistent? – Alka Mar 28 '21 at 15:17
Yes I tried to train it with six different datasets with different labels – roet Mar 28 '21 at 15:21
and what were the results for those trials ? it predicted always the same label "biodegradable"? – Alka Mar 28 '21 at 15:54
No it's shifting between different labels as I change the position of the image – roet Mar 28 '21 at 17:41

Connecting model to bounding boxes

1 Answers1