3

I'm currently using a resnet built in keras to do two class classification. I am using model checkpoint to save the best models based off of validation accuracy. Better and better models are saved until I go through all my datapoints a few times. Keras keeps saving new models showing they have higher accuracy but when I test the models they perform worse than previous models.

Here is an output of testing each model with validation data. The first number in the model name is the epoch, the last number is accuracy according to keras.

  • 03-0.90.hdf5, Wrong predictions on A: 176, Wrong predictions on B: 1652, total errors 1828
  • 04-0.91.hdf5, Wrong predictions on A: 246, Wrong predictions on B: 1448, total errors 1694
  • 06-0.92.hdf5, Wrong predictions on A: 135, Wrong predictions on B: 1738, total errors 1873
  • 09-0.92.hdf5, Wrong predictions on A: 117, Wrong predictions on B: 1738, total errors 1855
  • 10-0.92.hdf5, Wrong predictions on A: 183, Wrong predictions on B: 1208, total errors 1391
  • 15-0.92.hdf5, Wrong predictions on A: 64, Wrong predictions on B: 2973, total errors 3037
  • 23-0.93.hdf5, Wrong predictions on A: 119, Wrong predictions on B: 2511, total errors 2630
  • 26-0.93.hdf5, Wrong predictions on A: 113, Wrong predictions on B: 1794, total errors 1907
  • 27-0.93.hdf5, Wrong predictions on A: 85, Wrong predictions on B: 2821, total errors 2906

...

  • 1049-0.97.hdf5, Wrong predictions on A: 31, Wrong predictions on B: 5949, total errors 5980
  • 1089-0.97.hdf5, Wrong predictions on A: 3, Wrong predictions on B: 6137, total errors 6140
  • 1092-0.97.hdf5, Wrong predictions on A: 6, Wrong predictions on B: 6325, total errors 6331
  • 1095-0.97.hdf5, Wrong predictions on A: 0, Wrong predictions on B: 6408, total errors 6408
  • 1126-0.97.hdf5, Wrong predictions on A: 0, Wrong predictions on B: 6413, total errors 6413

As you can see, it seems the model is improving accuracy at predicting A even though accuracy on B is dropping more than A is improving.

Edit: Here is my code https://github.com/tan2684/Modified-Keras-Resnet

Cœur
  • 37,241
  • 25
  • 195
  • 267
user3029296
  • 113
  • 1
  • 2
  • 8
  • My data is basic 96x96 images of the outline of objects with a black background. The code is similar to https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html. The resnet is a modified version of https://github.com/raghakot/keras-resnet. If there is another important info for figuring out the problem please let me know – user3029296 Apr 23 '17 at 03:05
  • Wouldn't that mean your network is being trained to constantly predict A? You're reducing it to constant output, that's your hint. – Patrick Roberts Apr 23 '17 at 03:05
  • @PatrickRoberts What would I need to do to make it optimize for both A and B. I was under impression this should be training for both. When you look at the outputs after long training the network is just outputting the same thing regardless of input. – user3029296 Apr 23 '17 at 03:10
  • 1
    Please include your actual code, it could be that you are missing some input normalization that makes the results off. – Dr. Snoopy Apr 23 '17 at 03:56
  • @MatiasValdenegro I'll upload my code tomorrow when I get home. The network is basically the github I linked it my first comment. I removed the batch norm layers as it caused this issue to occur within a few epochs. (maybe cause my data is mostly black background?) – user3029296 Apr 23 '17 at 06:24
  • Try changing learning rate based on the error for each iteration – Sarthak Apr 24 '17 at 06:52
  • @Sarthak "based on the error for each iteration" im not sure what you mean. I tried lowering it but that didnt seem to fix it. Also I'm using a Nadam optimizer which has a adaptive learning rate. – user3029296 Apr 25 '17 at 02:31
  • If `keras` is saving the best model but in actual you are making wrong errors, that means only one thing which is that your dataset is imbalanced and accuracy isn't the true measure of goodness of your model. – enterML Dec 15 '17 at 11:26

0 Answers0