Boosting

Boosting is an ensemble learning technique that combines many weak learners —models only slightly better than chance— into a single strong predictor. Its defining trait is that it builds them in sequence: each new learner is trained to correct the errors of the ones before it, so the ensemble steadily concentrates on the hard cases.

The idea began as a theoretical question. In the late 1980s, Michael Kearns and Leslie Valiant asked whether an algorithm that only learns «weakly» —doing just better than a coin flip— could be turned into a «strong» learner of arbitrarily high accuracy. In 1990, Robert Schapire proved in «The Strength of Weak Learnability» that it could, and his constructive proof was the first boosting algorithm.

Weak learners, one after another

The mechanism is iterative. A simple model —often a shallow decision tree— is trained, the instances it misclassifies are identified, and those instances are given more weight in the next round. The final output is a weighted sum of all these models. Where a single tree may underfit (high bias), boosting reduces that bias by stacking many learners, each specialized in what the others get wrong.

From AdaBoost to gradient boosting

The first practical algorithm was AdaBoost, introduced by Yoav Freund and Robert Schapire in 1997. AdaBoost reweights the training examples after each iteration and weights each learner by its accuracy. In 2001, Jerome Friedman generalized it into gradient boosting: instead of reweighting examples, each new model is fit to the negative gradient of a chosen loss function, turning boosting into gradient descent in function space. This framing makes it possible to optimize very different objectives —regression, classification, ranking— within the same scheme.

Modern implementations

Gradient boosting spread through highly efficient libraries. XGBoost, by Tianqi Chen and Carlos Guestrin (2016), added regularization, second-order optimization, and native handling of missing data. LightGBM, from Microsoft (2017), sped up training with leaf-wise growth and gradient-based sampling. CatBoost, from Yandex (2018), tackled categorical variables directly and curbed a prediction bias of the classic method. All three still use decision trees as their weak learners.

Boosting versus bagging

Boosting should not be confused with bagging, the Random Forest family. Bagging trains many models in parallel on different samples and averages their votes to reduce variance; boosting chains them in series to reduce bias. That sequential dependence makes it more powerful but also more sensitive to noise and to overfitting when left unregularized. In 2026, and despite deep learning's dominance in text and images, gradient boosting remains the go-to tool for tabular data, where it often outperforms neural networks. Which method wins on any given problem is not settled: it depends on the size, noise, and structure of the data.