Bagging

Bagging is an ensemble learning technique that trains several versions of a model on different random samples of the training set and combines their predictions: by majority vote for classification, by averaging for regression. Its name is the acronym of Bootstrap AGGregatING. Leo Breiman proposed it in 1996, in the paper “Bagging Predictors.”

Bootstrap and aggregation

The method has two pieces. The bootstrap consists of generating several different training sets by sampling with replacement from the original: each sample may repeat some examples and omit others. On each of those sets an independent base model is trained. Then aggregation combines the outputs of all of them into a single prediction.

Why it works

The key is the instability of the base model. Bagging reduces variance—and with it overfitting—and is especially effective with models that change a lot in response to small variations in the data, such as deep decision trees. By contrast, it barely reduces bias: averaging many similar models stabilizes the prediction, but does not correct a base model that is systematically wrong.

Link to Random Forest

The most famous example of bagging is the Random Forest (Breiman, 2001), which applies bagging to decision trees and adds an extra dose of randomness: at each split of the tree, it considers only a random subset of the features. A useful byproduct of the bootstrap is the out-of-bag error: since each model leaves out, on average, about a third of the data (the ones not selected), those examples serve to estimate the error without needing a separate validation set.

Bagging versus boosting

It is worth not confusing bagging with the other great ensemble method, boosting. Bagging trains its models in parallel and independently, and attacks variance. Boosting trains them in sequence, each correcting the errors of the previous one, and attacks bias above all. They are complementary strategies, not interchangeable.