Random Forest

Random forest is an ensemble learning algorithm that builds many decision trees and combines their predictions—by majority vote for classification, by averaging for regression—to achieve a more accurate and robust model than a single tree. Leo Breiman introduced it in 2001, and it remains one of the go-to tools for tabular data.

Two doses of randomness

Its strength comes from introducing randomness twice over. First, each tree is trained on a different bootstrap sample of the data—a subset drawn at random with replacement; this is bagging. Second, at each split of a tree, only a random subset of the features is considered, not all of them. This double randomness makes the trees differ from one another—“decorrelated”—and that diversity is what gives the forest its power: many imperfect, varied trees, together, get it right more often.

Its advantages

Random forest brings together a notable set of virtues. It reduces overfitting compared with a single tree, by lowering variance; it is robust to noise and outliers; it handles high-dimensional data well; and it barely needs hyperparameter tuning to give good results. Moreover, it offers two practical gifts: a measure of feature importance and the out-of-bag error, an internal performance estimate that uses, for each tree, the data left out of its sample.

Its limits

It is not perfect. It is less interpretable than a single tree—a forest of hundreds of trees is a relatively black box—it is more costly in computation and memory, and its importance measures can be biased in favor of features with many distinct values or categories. Even so, its combination of accuracy, robustness and ease of use keeps it among the favorite algorithms.