CatBoost

CatBoost is an open-source library for gradient boosting on decision trees, developed by the company Yandex. Its name comes from “Categorical Boosting,” and its hallmark is handling categorical features natively and fighting a subtle kind of data leakage that affected earlier implementations. The original reference is the paper by Prokhorenkova and colleagues, “CatBoost: unbiased boosting with categorical features” (2018).

What it contributes

CatBoost introduces three ideas. The first is native handling of categorical features: there is no need to encode them by hand, because it converts them into numbers via target-based statistics. The second is ordered boosting, a variant that, to encode a category or compute an example's gradient, uses only the “earlier” examples in a random permutation, never the example's own label. The third is symmetric trees (oblivious), in which all nodes at the same level apply the same condition, which acts as regularization and speeds up inference. Primary source.

The problem it solves: target leakage

Target leakage occurs when, in building an example's features or training signal, its own target value is used indirectly. The typical case is encoding a category by the target's average including the row itself: the model looks very good in training and generalizes worse. CatBoost's “ordered” approach avoids it because neither the encoding nor an example's gradient ever looks at its own label.

When to choose it

CatBoost is designed for datasets with categorical features and provides defaults that reduce some initial setup. Comparing it with LightGBM or XGBoost requires measuring all three on the same data, budget and criterion. An honest nuance is in order: none of the three always wins; which is best depends on the dataset, the number of categorical features and the tuning budget. Primary source.