XGBoost

XGBoost (eXtreme Gradient Boosting) is an open-source library that implements gradient boosting of decision trees especially efficiently and at scale. Tianqi Chen and Carlos Guestrin introduced it in a paper at the 2016 KDD conference, although the project had started a couple of years earlier. It is one of the most used tools in machine learning on tabular data.

What gradient boosting is

Gradient boosting is an ensemble technique that builds the trees in sequence: each new tree is trained to correct the errors of the previous set of trees, fitting the gradient of the loss function. The final prediction sums those of all the trees. Unlike bagging, which averages independent models to reduce variance, boosting chains models to reduce bias.

Its innovations

What sets XGBoost apart is the engineering. It optimizes a regularized objective, which adds to the loss function a penalty for the tree's complexity to control overfitting. It incorporates native handling of missing values, a split-finding algorithm aware of the data's sparsity, tree pruning, parallelization and an efficient use of memory that lets it work with datasets that do not fit in RAM.

Why it became famous

XGBoost earned its fame in Kaggle competitions. Its own authors documented that, of the 29 winning solutions published in 2015, 17 used XGBoost. It thus became the first choice for structured-data problems, where it often beats more sophisticated models.

Its place in 2026

Today XGBoost shares the spotlight with two other gradient boosting libraries: LightGBM, prized for its speed on large sets, and CatBoost, skilled with categorical variables. Despite the push of deep learning, this trio remains the benchmark on tabular data, where tree boosting keeps a hard-to-match blend of accuracy, robustness and efficiency.