26

I have used LSTM from Keras to build a model that can detect if two questions on Stack overflow are duplicate or not. When I run the model I see the following output in the epochs.

Epoch 23/200
727722/727722 [==============================] - 67s - loss: 0.3167 - acc: 0.8557 - val_loss: 0.3473 - val_acc: 0.8418
Epoch 24/200
727722/727722 [==============================] - 67s - loss: 0.3152 - acc: 0.8573 - val_loss: 0.3497 - val_acc: 0.8404
Epoch 25/200
727722/727722 [==============================] - 67s - loss: 0.3136 - acc: 0.8581 - val_loss: 0.3518 - val_acc: 0.8391

I am trying to understand the meaning of each of these terms. Which of the above values is the accuracy of my model. I am comparatively new to machine learning, so any explanation would help.

Dookoto_Sea
  • 521
  • 1
  • 5
  • 16

2 Answers2

77

When training a machine learning model, one of the main things that you want to avoid would be overfitting. This is when your model fits the training data well, but it isn't able to generalize and make accurate predictions for data it hasn't seen before.

To find out if their model is overfitting, data scientists use a technique called cross-validation, where they split their data into two parts - the training set, and the validation set. The training set is used to train the model, while the validation set is only used to evaluate the model's performance.

Metrics on the training set let you see how your model is progressing in terms of its training, but it's metrics on the validation set that let you get a measure of the quality of your model - how well it's able to make new predictions based on data it hasn't seen before.

With this in mind, loss and acc are measures of loss and accuracy on the training set, while val_loss and val_acc are measures of loss and accuracy on the validation set.

At the moment your model has an accuracy of ~86% on the training set and ~84% on the validation set. This means that you can expect your model to perform with ~84% accuracy on new data.

I notice that as your epochs goes from 23 to 25, your acc metric increases, while your val_acc metric decreases. This means that your model is fitting the training set better, but is losing its ability to predict on new data, indicating that your model is starting to fit on noise and is beginning to overfit.

So that is a quick explanation on validation metrics and how to interpret them.

Christian Vincenzo Traina
  • 9,546
  • 2
  • 40
  • 63
Primusa
  • 13,136
  • 3
  • 33
  • 53
  • 1
    Thanks for the explanation, now I understand why the model stops training after 25 epochs approximately as the val_acc starts falling, which indicates that it starts over-fitting. – Dookoto_Sea Jul 15 '18 at 14:56
  • Appreciate @Primusa, for explaining the concepts in detail. – Here_2_learn Nov 15 '22 at 09:28
0

RNN (LSTM) is different from CNN, so there could be different causes. For LSTM model from Keras, I will modify two LSTM arguments;

  • Units: down
  • Stateful: false

In general, these are possibilities in neural network training and validation dataset accuracy difference. I gather answers from the Internet

Sourosh

  • Preprocessing (zero meaning, normalizing, etc.) to either your training set or validation set, but not the both
  • Some layers that perform differently during training and inference from scratch, your model might be incorrectly implemented
  • Overfitting

dk14

  • Not enough data-points, too much capacity
  • Ordering
  • No/wrong feature scaling/normalization
  • Learning rate too big
Cloud Cho
  • 1,594
  • 19
  • 22