Good accuracy but wrong prediction

Question

I have trained a model with a very good val_accuracy, but the predictions are completely wrong. Unfortunately answers to similar questions didn't help me. My network has a mutli label problem. The end result is to predict the 3 best labels for each picture from an album.

This is my function, which makes the output with the predictions.

def recognition(path):
    class = np.array(classes)
    for picture in os.listdir(path):
        pic = os.path.join(path, picture)
        pic = image.load_img(pic, size)
        pic = image.img_to_array(pic)
        pic = pic/255

This is my CNN:

model = models.Sequential()
model.add(pretrained)
model.add(layers.Dropout(0.3))
model.add(layers.Flatten())
model.add(layers.Dropout(0.3))
model.add(layers.Dense(256, activation = 'relu'))
model.add(layers.Dropout(0.3))
model.add(layers.Dense(7, activation = 'sigmoid'))

The val_acc is at 0.9527.

You are not explaining the problem, how exactly are the predictions wrong? — Dr. Snoopy, Oct 20 '19 at 17:35
The predictions I get for every picture don't match the label of Ground Truth at all. It gives me good values for the pictures, but with the completely wrong label. — glomba, Oct 20 '19 at 20:22
Thats not helpful, please show some ground truth labels versus predictions — Dr. Snoopy, Oct 20 '19 at 20:36

score 2 · Answer 1 · answered Oct 20 '19 at 22:00

2

For multi label problems, I don't think accuracy is a good metric. Try precision and recall. Keras Functional model giving high validation accuracy but incorrect prediction This discussion might be useful for you.

answered Oct 20 '19 at 22:00

Sree

973
2
14
32

score 0 · Answer 2 · answered Oct 20 '19 at 18:30

I don't think there's any issue with the sigmoid (provided that this is a multi-label task). I would advice you to look at your data, try to make sense of whats happening. Maybe your dataset is completely different from the real world data (look into covariate shift). Just make sure you are not leaking any information into the validation set. There's no easy way to debug what is happening if the issue is data related.

score 0 · Answer 3 · answered Feb 03 '22 at 16:03

Note that the accuracy may be deceptive. One case is when the data is imbalanced. Assume there are a total of 600 samples, where 550 belong to the Positive class and just 50 to the Negative class. Since most of the samples belong to one class, the accuracy for that class will be higher than for the other. If the model made a total of 530/550 correct predictions for the Positive class, compared to just 5/50 for the Negative class, then the total accuracy is (530 + 5) / 600 = 0.8917. This means the model is 89.17% accurate. With that in mind, you might think that for any sample (regardless of its class) the model is likely to make a correct prediction 89.17% of the time. This is not valid, especially when you consider the Negative class for which the model performed badly.

The precision is calculated as the ratio between the number of Positive samples correctly classified to the total number of samples classified. magine a man who is trusted by others; when he predicts something, others believe him. The precision is like this man. When the precision is high, you can trust the model when it predicts a sample as Positive. Thus, the precision helps to know how the model is accurate when it says that a sample is Positive.

reference: Evaluating Deep Learning Models

Good accuracy but wrong prediction

3 Answers3