Model Ensemble

A model ensemble combines the predictions of several models into a single answer that beats any one of them on its own. The intuition is old and simple: where one model hesitates, another gets it right, and the joint decision is more reliable than that of a single expert.

The reason it works is statistical and precise. If the individual models are reasonably good and, above all, their errors are diverse —weakly correlated with one another— then combining them lets those errors cancel out while the shared signal reinforces. Pooling copies of the same model that always fail in the same way buys you nothing: diversity is the essential ingredient. In the language of the bias-variance trade-off, a well-built ensemble lowers the error without demanding a more complex base model.

Bagging: averaging in parallel

Bagging (short for bootstrap aggregating, formalized by Leo Breiman in 1996) trains many models in parallel, each on a resampling of the data drawn with replacement, and then averages or votes their outputs. Its main effect is to reduce variance: it stabilizes methods that, trained on slightly different data, would otherwise give wildly different answers. The Random Forest, also due to Breiman, is its flagship case —hundreds of decision trees, each grown on a different sample and a random subset of features—; it is covered in the Decision Trees entry.

Boosting: correcting in sequence

Boosting flips the recipe: it trains models in sequence, and each one focuses on the errors left by the previous ones. AdaBoost, by Yoav Freund and Robert Schapire, was the first to make it popular; Jerome Friedman's gradient boosting generalized it by fitting each new model to the gradient of the error. Its main effect is to reduce bias. This family gave rise to XGBoost, LightGBM and CatBoost, which today dominate problems with tabular data and have their own entry, Boosting.

Stacking, voting and limits

Beyond these two families, stacking trains a meta-model that learns to combine the outputs of several heterogeneous base models, while voting or plain averaging aggregate them by majority or mean. No technique is free: an ensemble multiplies the cost of computation and inference —many models are trained and run instead of one— and sacrifices interpretability, because the decision no longer lives in a single readable model. The famous Netflix Prize, whose winning solution was an ensemble of numerous algorithms, illustrates both the method's power and its practical complexity. The sensible rule is clear: reach for an ensemble when the gain in accuracy justifies that cost, not by default.