1

in the video Anna Veronika Dorogush - CatBoost - the new generation of Gradient Boosting

she talks in minute 15:30 about "objects". What exactly does she mean? Features? Columns? Rows?

And what exactly is the difference between classical and ordered boosting? Does classical just add up all decision trees? Does the ordered use just the decision trees that were randomly distributed before and adds them up? So the difference is just the random distribution?

I hope someone can help me because I have a really hard time understanding this.

Kamel
  • 141
  • 1
  • 9

2 Answers2

3

To my understanding, what she mentioned as "objects" in the video are the data points/instances of the dataset. The problem of classical boosting mentioned in the catboost paper is prediction shift. That is, what the model learns in the training set does not reflect in the testing set. They say that the root of the problem is that each tree in the training phase is trained on the same set of data points, thus having no chances of experiencing unseen data.

For ordered boosting, a tree is trained on a subset of the data set and used to calculated residuals for another subset that it hasn't seen. Catboost obtains this by creating an artificial time, that is, a random permutation of the data.

Let's say you have ten data points from 0 to 9. Catboost will create a permutation that contains 5,0,2,1,3,6,4,9,7,8 (this is just an arbitrary permutation I came up with), and a model is trained on 5,0,2,1,3 and then used to compute residuals of 6,4,9,7,8.

This is just my own understanding, and by no means, I say it's 100% right. Any comments and corrections are much welcome and appreciated.

hamnghi
  • 99
  • 1
  • 7
0

By watching the rest of the video it is clear that when she uses the term "object," she is actually referring to a specific row within the dataset. (It is obvious when she talks about the SHAP values of an object at 23:00)

meowlicious
  • 11
  • 1