LightGBM

LightGBM (Light Gradient Boosting Machine) is an open-source gradient boosting library built on decision trees, developed by Microsoft. It was introduced by Guolin Ke and colleagues in “LightGBM: A Highly Efficient Gradient Boosting Decision Tree”, published at NeurIPS 2017. Its promise is precise: train boosting models as accurate as the classic ones, but far faster and with a much smaller memory footprint.

Like any boosting method, LightGBM builds decision trees sequentially, each one correcting the errors of the last; we cover that idea in Boosting and in Ensemble of Models. What sets LightGBM apart is how it reaches that efficiency.

The optimizations: histograms, leaf-wise, GOSS and EFB

The first ingredient is its use of histograms: instead of scanning every continuous value of a feature, LightGBM groups them into discrete «buckets» (bins). Finding the best split then depends on the number of bins rather than the number of data points, which speeds up computation and cuts memory use. Primary source.

The second is leaf-wise tree growth: at each step it splits the leaf that promises the largest loss reduction, instead of expanding the tree level by level. The result is asymmetric trees that tend to reach lower loss with fewer splits. Primary source.

On top of this come two techniques introduced in the paper. GOSS (Gradient-based One-Side Sampling) keeps the instances with the largest gradients —the most informative ones— and discards much of the small-gradient data, estimating information gain from far fewer examples. EFB (Exclusive Feature Bundling) bundles sparse, mutually exclusive features —those rarely nonzero at the same time— into a single one, reducing dimensionality with almost no loss of information. Primary source.

Strengths and the overfitting risk

Together, these ideas make LightGBM one of the fastest and lightest options for large, high-dimensional data: according to Ke et al., it speeds up conventional GBDT training by more than twenty times while keeping almost the same accuracy. That is why it is used for tabular classification, regression and ranking tasks. Primary source. Primary source. Primary source.

Against XGBoost and CatBoost

LightGBM shares ground with two other gradient boosting libraries. XGBoost is the mature, robust reference; it traditionally grows trees level-wise, which makes it very stable but, on massive datasets, often slower and more memory-hungry than LightGBM. CatBoost, from Yandex, stands out for its native handling of categorical features and its symmetric trees, which curb overfitting at some cost in speed. There is no universal winner: LightGBM shines when data volume and training speed matter most, while its alternatives may suit small sets or many categorical variables. Test and validate before deciding. Documentation: original LightGBM paper; original XGBoost paper; original CatBoost paper.