5

I am using H2O via R. I am trying to build random forest, XGBoost, GBM models to solve multiclass problem.

The model performance insights that H2O provides are great but as one of the success criterias I have my own custom function that scores the model accuracy when model is used to score a set of users say, validation set. Generally speaking, this function rewards the prediction and punishes the mis-prediction. Given that the target classes in this problem are ordinal categories, the punishment score depends on the number of places by which prediction was missed. The inverse of this custom function can also be thought of as a loss function.

I am wondering if and how I can plug such custom loss function into H2O while training models.

desertnaut
  • 57,590
  • 26
  • 140
  • 166
sarang
  • 51
  • 3

1 Answers1

6

While there currently isn't support for a custom loss function (though there is a jira ticket to add it in), there is some support for a custom evaluation function, you can find more details here: https://github.com/h2oai/h2o-3/blob/master/h2o-docs/src/dev/custom_functions.md.

Lauren
  • 5,640
  • 1
  • 13
  • 19
  • 3
    Thanks Lauren. Are custom evaluation functions used only for reporting or can they also affect early stopping (as stopping_metric)? – sarang Sep 07 '18 at 22:04
  • 1
    great follow up question! you can find the answer here: https://stackoverflow.com/questions/51657527/implementing-custom-stopping-metrics-to-optimize-during-training-in-h2o-model-di. The short answer is not at the moment but there is a jira ticket for this feature request. – Lauren Sep 07 '18 at 22:43
  • FYI this has been added in `3.26.0.1`, released in July 2019 – nirvana-msu Aug 11 '21 at 11:53