1

It seems that the sum of corresponding leaf values of each tree doesn't equal to the prediction. Here is a sample code:

X = pd.DataFrame({'x': np.linspace(-10, 10, 10)})
y = X['x'] * 2
model = xgb.XGBRegressor(booster='gbtree', tree_method='exact', n_estimators=100, max_depth=1).fit(X, y)
Xtest = pd.DataFrame({'x': np.linspace(-20, 20, 101)})
Ytest = model.predict(Xtest)
plt.plot(X['x'], y, 'b.-')
plt.plot(Xtest['x'], Ytest, 'r.')

enter image description here

The tree dumps reads:

model.get_booster().get_dump()[:2]

['0:[x<0] yes=1,no=2,missing=1\n\t1:leaf=-2.90277791\n\t2:leaf=2.65277767\n',
 '0:[x<2.22222233] yes=1,no=2,missing=1\n\t1:leaf=-1.90595233\n\t2:leaf=2.44333339\n']

If I only use one tree to do prediction:

Ytest2 = model.predict(Xtest, ntree_limit=1)
plt.plot(XX1['x'], Ytest2, '.')
np.unique(Ytest2)  # array([-2.4028,  3.1528], dtype=float32)

enter image description here

Clearly, Ytest2's unique values does not corresponds to the leaf value of the first tree, which is -2.90277791 and 2.65277767, although the observed split point is right at 0.

  1. How are the leaf values related to the predictions?
  2. Why are the leaf values in the first tree not symmetric, provided that the input is symmetric?
doraemon
  • 2,296
  • 1
  • 17
  • 36

1 Answers1

2

Before fitting the first tree, xgboost makes an initial prediction. This is controlled by the parameter base_score, which defaults to 0.5. And indeed, -2.902777 + 0.5 ~=-2.4028 and 2.652777 + 0.5 ~= 3.1528.

That also explains your second question: the differences from that initial prediction are not symmetric. If you set learning_rate=1 you probably could get the predictions to be symmetric after one round, or you could just set base_score=0.

Ben Reiniger
  • 10,517
  • 3
  • 16
  • 29
  • I actually tested `model.base_score` which returns `None`. How to get the `base_score` used? – doraemon Jun 16 '21 at 23:46
  • xgb is a little weird in setting the python defaults always to `None`, then setting the real default value in its lower-level code. I haven't found where it does that yet, but the docs do advertise the default as 0.5, and the arithmetic checks out in your case. (Can you call `predict` with `n_tree_limit=0`?) – Ben Reiniger Jun 18 '21 at 13:50