Few-Shot Learning

Few-shot learning aims to have a model learn a new task from very few labeled examples, usually a handful per class. Its extreme cases are learning from a single example per class (one-shot) and learning with no example of the target class at all (zero-shot), which generalizes from descriptions or attributes. With so little data, the model must rely on prior knowledge.

Two senses that are not the same

Two uses coexist today and are worth distinguishing. In the classic sense, tied to meta-learning, a model is trained over many few-example tasks so that it learns to generalize from little data; here the examples adjust the weights. In the foundation-model sense, “in-context learning” consists of giving a few examples inside the prompt and letting the model solve the task without updating any weights; it was popularized by GPT-3 (Brown et al., 2020), whose paper is actually titled “Language Models are Few-Shot Learners.” In the first case the examples train; in the second they are only shown.

Methods of the classic approach

The classic approach rests on meta-learning, or “learning to learn.” Two families stand out: metric-based methods, such as prototypical networks (Snell et al., 2017), which classify by closeness to a prototype of each class; and optimization-based methods, such as MAML (Finn et al., 2017), which learns initial weights such that a few adjustment steps suffice to adapt to a new task.

Limits

In-context learning is very sensitive to the chosen examples and their order: changing only the order of the demonstrations in the prompt can shift the result by tens of points. Moreover, that capability is not “learning” in the classic sense, but an emergent property of massive pre-training: without that pre-training it would not exist. And meta-learning methods can be fragile when the new tasks stray from those seen during meta-training.