Which are the differences between setting objective='binary:logistic' and objective='binary:logitraw' in a xgboost classifier?
Based on the documentation (https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters), the former corresponds to logistic regression for binary classification, output probability, while the latter is logistic regression for binary classification, output score before logistic transformation.
It is not clear to me what these mean in practice. Could you explain which functions are minimized during training for the two cases?
Also it appears that setting the objective parameter does not change the model output at all, as in the code below.
Simulate data:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import xgboost as xgb
x1 = np.random.uniform(low=-3,high=4,size=10000)
x2 = np.random.uniform(low=-3,high=4,size=10000)
x3 = np.random.uniform(low=-3,high=4,size=10000)
X = pd.DataFrame({'x1':x1, 'x2':x2, 'x3':x3})
z = 2 * x1 + 3 * x2 + 4 * x3
def invlogit(z):
p = 1 / (1 + np.exp(- z))
return p
pr = invlogit(z)
y = pd.Series(data=np.random.binomial(size=10000, n=1, p=pr))
Define two classifiers, all parameters are the same except objective:
params={'gamma': 1.4,
'learning_rate': 0.2,
'max_delta_step': 5.,
'max_depth': 8,
'min_child_weight': 2.2,
'subsample': 0.7,
'objective':'binary:logistic',
'nthread':4,
'seed':2,
'num_boost_round':200,
'reg_alpha':0,
'reg_lambda':0
}
clf = xgb.XGBClassifier(**params)
clf.fit(X, y)
tmp1=params.copy()
tmp1['objective']='binary:logitraw'
clf1=xgb.XGBClassifier(**tmp1)
clf1.fit(X, y)
Plot predictions (invlogit is the inverse of a logit function and gives a probability).
plt.plot(invlogit(clf.predict(X, output_margin=True)),
invlogit(clf1.predict(X, output_margin=True)),'.')
plt.xlabel('binary:logistic');
plt.ylabel('binary:logitraw');
I'm confused, the results are the same for both cases. Any guess? Thanks!
