1

I use the seq2seq model and it can compute BLEU score (a NMT score) every epoch. However, I cannot set BLEU score as validation metric so it cannot early stop in training. I read the source code, but there are no hints as to what kind of string could be added to the validation metrics except for "+loss". Please save me, thanks!

qiuyuan
  • 189
  • 1
  • 4

1 Answers1

2

The default validation_metric is actually "-loss", not "+loss". The "-" means this is a metric that should be minimized, not maximized.

So to use BLEU score instead, set the validation_metric to "+BLEU".

In general, you can use any metric that's returned by your model's .get_metric() method. The name of the metric you use for validation_metric just has to match the corresponding key from the dictionary returned by .get_metric().

In your case, presumably your model's .get_metric() method returns something like this: {"BLEU": ...}, which is why validation_metric should be set to "+BLEU".

petew
  • 671
  • 8
  • 13
  • Thank you very much! It solve my problem! Since my problem is why "BLEU" instead of "bleu" (i.e., how the str is related to paticular metric), I have two suggestions to the document: It would be better if it is commented to the parameter "validation metric", or log the available metric of "validation_metric". It would very helpful for the green-hand just like me, as I read and search the available "metric str" by reading source code for two hours but failed :) – qiuyuan Oct 27 '20 at 01:27
  • @qiuyuan, would you be interested in making a PR to improve the documentation on this? – petew Oct 27 '20 at 16:24
  • 1
    Of course, I will have a try and add some complementary documentation recently :) – qiuyuan Oct 28 '20 at 00:59