How to perform linear/logistic regression on predictions of different models (say randomforest, gbm, svm etc)?

Question

Basically it is done to improve the predictions by creating an ensemble. But how do we do that. Could somebody please explain using a sample code in R? I am just a learner. Any help would greatly be appreciated.

Thank you.

Answer 1

Prediction aggregation in ensembles can be done in a large variety of ways. The simplest approach is majority voting (classification) or averaging the predictions of all base models (regression).

Often, complex aggregation schemes are not much better than the basic ones (and are very sensitive to overfitting). This is why specialized packages such as EnsembleSVM only permit very basic aggregation (a linear combination at best).