Ensemble Learning

Ensemble learning is the technique of combining the predictions of several models—the “base learners”—to seek a result that is more accurate or robust than a single predictor. Improvement is not automatic: it depends on the models, their errors, and the rule used to combine them.

Why it works

Diversity can help, but it is not sufficient by itself. In 1994, Krogh and Vedelsby showed, for ensembles of networks predicting continuous values, that the error of the average equals the members’ mean error minus an “ambiguity” term measuring their disagreement. Models with different errors can therefore improve the ensemble; the benefit depends on the quality of each component and on how they are combined, not on a guarantee of beating every individual model.

The three methods

There are three main strategies. Bagging (Leo Breiman, 1996) trains the models in parallel on random samples of the data and averages their outputs; it reduces variance, and its most famous example is the Random Forest. AdaBoost, introduced by Freund and Schapire, builds a sequence that gives relatively more weight in the next round to examples the current classifier gets wrong; XGBoost, described by Chen and Guestrin in 2016, is a scalable tree boosting system. Wolpert’s stacking uses the predictions of base generalizers as inputs to another generalizer that learns how to combine them.

How they are combined

The simplest combination is voting: in classification, the most-voted class among the models; in regression, the average of their predictions. Boosting usually uses a weighted version, giving more weight to the more accurate models.

Advantages and limits

Its potential advantages include greater accuracy, robustness and generalization ability, provided that the models and combination rule are appropriate. But there are also costs. Interpretability suffers—it is hard to explain a prediction that is the average of many models—the computational cost of training and maintaining the set increases, and implementation, especially of stacking, becomes more complex.