0

I am training a multi-lable classification model on keras, after 10 epochs it's showing 91.80% accuracy on training set and 86.95% on validation set. When I am testing the same training set using model's predict method I am getting different(78%) accuracy. Is it because keras is using different method for computing accuracy?

Any ideas will be appreciated on how to overcome this difference.

Please find training and testing code @GithubGist

Epoch 1/10
2019-05-28 20:50:45.675999: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-05-28 20:50:45.840580: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
9600/9600 [==============================] - 99s 10ms/step - loss: 0.5300 - acc: 0.7443 - val_loss: 0.4317 - val_acc: 0.7950
Epoch 2/10
9600/9600 [==============================] - 60s 6ms/step - loss: 0.4399 - acc: 0.7911 - val_loss: 0.3859 - val_acc: 0.8305
Epoch 3/10
9600/9600 [==============================] - 60s 6ms/step - loss: 0.3891 - acc: 0.8256 - val_loss: 0.3607 - val_acc: 0.8485
Epoch 4/10
9600/9600 [==============================] - 60s 6ms/step - loss: 0.3520 - acc: 0.8466 - val_loss: 0.3182 - val_acc: 0.8455
Epoch 5/10
9600/9600 [==============================] - 59s 6ms/step - loss: 0.3225 - acc: 0.8609 - val_loss: 0.3229 - val_acc: 0.8705
Epoch 6/10
9600/9600 [==============================] - 59s 6ms/step - loss: 0.2920 - acc: 0.8788 - val_loss: 0.3143 - val_acc: 0.8640
Epoch 7/10
9600/9600 [==============================] - 60s 6ms/step - loss: 0.2644 - acc: 0.8903 - val_loss: 0.3308 - val_acc: 0.8645
Epoch 8/10
9600/9600 [==============================] - 62s 6ms/step - loss: 0.2397 - acc: 0.9030 - val_loss: 0.3322 - val_acc: 0.8725
Epoch 9/10
9600/9600 [==============================] - 60s 6ms/step - loss: 0.2226 - acc: 0.9108 - val_loss: 0.3353 - val_acc: 0.8670
Epoch 10/10
9600/9600 [==============================] - 60s 6ms/step - loss: 0.2029 - acc: 0.9180 - val_loss: 0.3797 - val_acc: 0.8695
danishansari
  • 634
  • 5
  • 21
  • Usually that's what happens. Your training accuracy will be higher than your testing and validation accuracy. 78% doesn't mean it has over fitted, I think, but still you can use Dropouts, Batch Normalization type of things to reduce the possibility. Not sure about your data, but you can also eliminate the least contributing features, or you might want to reduce the complexity of your neural network. – Mansur May 28 '19 at 15:52
  • @MensurQulami problem is the accuracy is different on the same training dataset, not on testing or validation. – danishansari May 28 '19 at 15:53
  • in my guess this can only happen if you apply different pre-proessing as the data is entirely same, but if you check out my code, I am definitely doing the same pre-processing. – danishansari May 28 '19 at 15:55
  • Not sure, but it might be because of the `batch_size`. I mean while training it trains in batches, however it is testing 1 by 1 in your case. I don't say it is the very case, though. – Mansur May 28 '19 at 16:01
  • batching just mean parallel processing, so I don't think that can be the issue, anyways thank you, – danishansari May 28 '19 at 16:05
  • Are you sure this line ```predn = np.array([0 if e < 0.46 else 1 for e in predn[0]])``` is not creating the difference? – Anakin May 28 '19 at 18:03
  • Also, if you have `Dropout` and `BatchNormalization` layers, they might cause some deviation; but I would expect that to be less than in your case. – Anakin May 28 '19 at 18:07
  • About getting prediction from sigmoid output can be problematic, but I checked it with values bet 0.2 and 0.7 at 0.45 it's giving the best accuracy. What do you think can be done to get the predictions out of sigmoid activation. – danishansari May 28 '19 at 18:57
  • I am not using batch norm, and I don't think dropout can cause that much of a change in prediction. – danishansari May 28 '19 at 18:58
  • keras calculates `binary_crossentropy` accuracy as `K.mean(K.equal(y_true, K.round(y_pred)))` i.e. the threshold is at `0.5`. In your case, it is `0.45`. That difference might explain the deviation. I would suggest you to set the threshold at `0.5` and check if you are still getting a large deviation from the training. – Anakin May 29 '19 at 08:30
  • I check it on values between 0.2 to 0.7, 0.45 is giving the best not 0.5. anyways i think i got it, keras is computing accuracy on each label being correct, while i was computing on each image being correctly classified, as its a multi-lable problem. – danishansari May 30 '19 at 19:37

0 Answers0