XGBoost: Early stopping on default metric, not customized evaluation function

Question

I am using XGBoost 0.90. I wish to train a XGBoost regression models, with Python, using a built-in learning objective with early stopping on a built-in evaluation metric. Easy. In my case the objective is 'reg:tweedie' and the evaluation metric is 'tweedie-nloglik'. But at each iteration I also wish to calculate an informative custom metric, which should not be used for early stopping. But it wrongly is.

Eventually I wish to use scikit-learn GridSearchCV, train a a set of models to early stopping with built-in objectives and metrics, but at the end choosing that which does best on a custom metric over the folds.

In this sample code I am using another built-in objective and built-in metric, but the problem is the same.

import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.model_selection import train_test_split

def mymetric(pred, dmat):
    y = dmat.get_label()
    res = np.sqrt(np.sum((y - pred)**4)/len(y))
    return 'mymetric', float(res)

np.random.seed(seed=2500)
x, y, weight = np.random.randn(4096, 16), np.random.randn(4096), np.random.random(4096)

train_x, test_x, train_y, test_y, train_weight, test_weight = train_test_split(x, y, weight, 
                                                                               train_size=0.7, random_state=32)

dtrain = xgb.DMatrix(train_x, label=train_y, weight=train_weight)
dtest = xgb.DMatrix(test_x, label=test_y, weight=test_weight)

results_learning = {}
bst = xgb.train(params={'objective': 'reg:squarederror', 
                        'eval_metric': 'rmse', 
                        'disable_default_eval_metric': 0},
    num_boost_round=20, dtrain=dtrain, evals=[(dtrain, 'dtrain'), (dtest, 'dtest')],
    evals_result=results_learning,
    feval=mymetric,
    early_stopping_rounds=3)

The output is (it would have stopped at iteration 3 if I had not used feval):

[0] dtrain-rmse:1.02988 dtest-rmse:1.11216  dtrain-mymetric:1.85777 dtest-mymetric:2.15138
Multiple eval metrics have been passed: 'dtest-mymetric' will be used for early stopping.
Will train until dtest-mymetric hasn't improved in 3 rounds.
...
Stopping. Best iteration:
[4] dtrain-rmse:0.919674    dtest-rmse:1.08358  dtrain-mymetric:1.56446 dtest-mymetric:1.9885

How could I get an output like this?

[0] dtrain-rmse:1.02988 dtest-rmse:1.11216  dtrain-mymetric:1.85777 dtest-mymetric:2.15138
Multiple eval metrics have been passed: 'dtest-rmse' will be used for early stopping.
Will train until dtest-rmse hasn't improved in 3 rounds.
...
Stopping. Best iteration:
[3] dtrain-rmse:0.941712    dtest-rmse:1.0821   dtrain-mymetric:1.61367 dtest-mymetric:1.99428

I could have solved this with a custom evaluation function returning a list of tuples (https://github.com/dmlc/xgboost/issues/1125). But can this done when I wish to use built-in evaluation metrics like 'rmse' or 'tweedie-nloglik'? Can I call them inside the custom evaluation function?

score 2 · Answer 1 · answered Jan 26 '21 at 21:50

There is a built-in early stopping callback function in XGBoost in which it's possible to specify which dataset and which metric to use for early stopping. In your case, you'd have to create a new early stopping callback like this:

early_stop = xgb.callback.EarlyStopping(rounds=3,
                                    metric_name='rmse',
                                    data_name='dtest')

And then add it to the list of callbacks when you call train:

bst = xgb.train(params={'objective': 'reg:squarederror', 
                    'eval_metric': 'rmse', 
                    'disable_default_eval_metric': 0},
num_boost_round=20, dtrain=dtrain, evals=[(dtrain, 'dtrain'), (dtest, 'dtest')],
evals_result=results_learning,
feval=mymetric,
callbacks=[early_stop])

See this page of the docs for more information: https://xgboost.readthedocs.io/en/latest/python/callbacks.html

XGBoost: Early stopping on default metric, not customized evaluation function

1 Answers1