2

I am studying transformer model with this tutorial and I am confused on the difference between evaluation and inference. In my understanding evaluation happens after the model is trained, by only giving it source and ask it to predict the target one by one (in the seq2seq problem).

However, in the tutorial it does evaluation in the same way as training, which is getting the loss from a forward pass to the model, given both source and target. And the inference step is more similar to what I understand as evaluation. In this case, I tried the model and it does really well with evaluation and testing, but at the inference step I found that it can't output anything meaningful. Can anyone explain me the difference between evaluation and inference?

Natalia
  • 153
  • 1
  • 8

1 Answers1

1

Evaluation and inference are indeed two different subjects, but the former can be performed only by means of the latter. Practically, you repeatedly make an inferences in order to perform the evaluation.

At this specific step:

 train_loss = train(model, train_iterator, optimizer, criterion, CLIP)
 valid_loss = evaluate(model, valid_iterator, criterion)

After each epoch on the training set, we need to evaluate/make an inference on each sample from the validation set, in order to check if we have overfitting or underfitting or other phenomena. This happens in any robust machine learning process (at least it should happen).

If you have problems when testing, it can be due to two different reasons:

  1. You either do not test the same way you fed the input to the training + validation.
  2. The test set is so different from the training and the validation that your model does not behave well when subjected to inputs statistically different from the training and the validation you used.

Also, note that the evaluation and inference is not related to seq2seq/nor is it different in other machine learning problems as compared to seq2seq; the same concepts of evaluation and inference apply for all machine learning tasks.

Timbus Calin
  • 13,809
  • 5
  • 41
  • 59
  • Thank you for your answer! This is the same as my understanding, but what confuses me in this tutorial is that there is another inference function which is done differently from the evaluation. In the tutorial, testing, validating and training are all done in the same way, but inference is not. So the model performed quite coherently on testing, but not well in inference. – Natalia Jun 29 '20 at 11:48