0

Can someone compare and contrast these two concepts in layman terms for me? The definitions sound similar but I know there has to be more differences between the two.

I have:

  • Ensemble models: Combines multiple ML models together to get a better model.
  • Boosting: Improving a single weak model by combining it with a number of other weak models in order to generate a collectively strong model.
Katsu
  • 8,479
  • 3
  • 15
  • 16
  • I thought boosting meant an iterative approach: residual errors from prior model were fed into subsequent model to reduce them further. I think of the errors as a new input step to an iterative process that drives the errors closer to zero. – duffymo Jul 29 '22 at 12:30

2 Answers2

1

Ensemble is a weighted combination of several models that returns a single result. The weights can be thought of as a measure of your confidence in each model relative to the others.

I thought boosting meant an iterative approach: residual errors from prior model were fed into subsequent model to reduce them further. I think of the errors as a new input step to an iterative process that drives the errors closer to zero.

duffymo
  • 305,152
  • 44
  • 369
  • 561
  • One follow up question: in that boosting iterative process, are bad models weighted more or less? I thought bad models were weighted more so that the model focuses its attention on improving those bad models more. Ive also seen elsewhere that bad models are weighted less because they are "bad" and we want to give more weight to good models. Thoughts? – Katsu Jul 29 '22 at 20:54
  • 1
    The weight is used to combine the results of models to come up with the final result. I think the weights represent confidence in the model, values between 0 and 1 and sum to 1.0. A higher weight says "I believe this model more than the others. I'd like it to count for more in my final result." The limit would be 1.0 weight for a single model and zeroes for all the others. – duffymo Jul 29 '22 at 23:55
  • So to give a concrete example: Lets say i have two trees in the ensemble model. After 1 iteration of training, we see that one is more correct and one is more incorrect. Would the respective weights be 0.75 for the more correct one and 0.25 for the more incorrect one (not assigning 1 and 0 because the model hasnt completely finished training yet)? And after training has completed, the incorrect models would be weighted as 0 and correct models weighted as 1? – Katsu Oct 06 '22 at 21:20
0

To elaborate on @duffymo;

Ensemble simply means "collection" so it just a collection of different models (or the same) - think of Random Forest. It is a collection of (different) Decision Trees where we then average the outputs from them to create 1 "meta" model.

I would say that boosting is an ensemble, but created in a specific way. Different boosting algorithms do it differently but what they have in common is, that they use the errors from the previous model, to create a better model in the next step. One way of creating a boosting algorithm would be:

  1. Fit some baseline model,m_0 (regression could be the mean of y_train)
  2. Calculate the error/residuals,e, for y_train using the model M = m_0
  3. Fit a model (that could be a Linear Regression) m_1 to predict e
  4. Create a new model as M = m_0+m_1
  5. Repeat (2)-(4) as many times you want, such that your model is M=m_0+m_1+m_2...

Why does this work?

Since the error e is defined as e = y_train-m_0(x) (where m_0(x) is the predictions using m_0) then we can train a model, m_1 to predict e i.e we can approximate e by m_1(x) thus we then get

m_1(x)=y_train-m_0(x) which then implies y_train = m_1(x)+m_0(x) (our model in step (4)). That model is not perfect thus we can iterate over it again and again by adding a new model that fits the residual of the previous M.

Some algorithms, like XGBoost would add a "learning rate",alpha to each of the models, such that M = m_0 + alpha*m_1+alpha*m_2...

but that's another story

CutePoison
  • 4,679
  • 5
  • 28
  • 63