Train and validation score is high but very Poor Test Accuracy

Question

I am working on multi-label image classification, i am using inception net as my base architecture. after the complete training i am getting, training accuracy > 90% and validation accuracy > 85% but i am getting 17% accuracy on test data.

Model training -->

model = Model(pre_trained_model.input, x)
model.compile(loss='categorical_crossentropy',
              optimizer=RMSprop(lr=0.0001),#'adam'
              metrics=['acc'])
    history = model.fit_generator(
      train_generator,
      steps_per_epoch=600,#total data/batch size
      epochs=100,
      validation_data=validation_generator,
      validation_steps=20,
      verbose=1,callbacks = callbacks)

Testing on the trained model:

test_generator = test_datagen.flow_from_directory(
    test_dir,target_size=(128, 128),batch_size=1,class_mode='categorical')

filenames = test_generator.filenames
nb_samples = len(filenames)

prediction = test_model.predict_generator(test_generator,steps=nb_samples,verbose=1)

Saving the results to Pandas

predicted_class_indices = np.argmax(prediction,axis=1)
labels = (train_generator.class_indices) #geting names of classes from folder structure
labels = dict((v,k) for k,v in labels.items())
predictions = [k for k in predicted_class_indices]

results=pd.DataFrame({"image_name":filenames,
                      "label":predictions})
results['image_name'] = [each.split("\\")[-1] for each in results['image_name']]

Everything looks fine but still i am getting very poor prediction. kidly help me to fugure out, where i am making the mistakes.

score 0 · Answer 1 · answered Feb 15 '19 at 07:30

It can be the case that the images in your dataset are arranged in such a way that test images are previously unseen by the model and so the accuracy drops significantly.

What I recommend is for you to try to use K-fold cross validation or even Stratified K-fold cross validation. The benefit here is that your dataset will be splitted in, let's say 10 'batches'. Every iteration (out of 10) one batch will be the test batch and all the others will be train batches. The next iteration, test batch from the previous step becomes train batch and some other batch becomes test batch. It's important to denote that every batch will be the test batch only once. Another benefit of the Stratified K-fold is that it will take into account the class labels and try to split the classes in such way that every batch has approximately the same distribution of classes.

Another way to achieve some better results is to just shuffle the images and pick the training ones and test ones then.

Train and validation score is high but very Poor Test Accuracy

1 Answers1