Active Learning

Active learning is a machine-learning approach in which the algorithm chooses which examples it wants to learn from, querying the most informative labels from an “oracle”—usually a human annotator—instead of passively receiving an already-labeled dataset. Its goal is to reach good accuracy with far fewer labels, reducing the cost of annotation, and it rests on a simple idea: not all examples are equally informative.

The loop

It works as a human-in-the-loop cycle. A model is trained on a few labeled data; the model reviews a large unlabeled set and selects the most useful examples; the oracle labels them; the model is retrained with them; and it repeats until the budget is exhausted or the desired performance is reached. The key is that the model itself directs where the human effort is concentrated.

How it chooses what to ask

The most common strategy is uncertainty sampling: asking for the labels of the examples the model is least sure about, typically those near the decision boundary. Other strategies are query by committee—choosing what several models disagree on most—and diversity or representativeness criteria, which avoid chasing only outliers.

Relation and a risk

Active learning should be distinguished from semi-supervised learning. Settles's report explains that both use unlabeled data from complementary directions: semi-supervised learning incorporates predictions for examples the model considers most certain, whereas active learning requests labels for those it is least certain about. The two approaches can be combined. Their central methodological risk is sampling bias: because the model chooses what to label, the queried set may cease to represent the original distribution and can bias evaluation if it is treated as a random sample.