Curriculum Learning

Curriculum learning is a training strategy that feeds a model its examples in a meaningful order —typically from easiest to hardest— rather than sampling them at random. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston formalized and named this strategy at ICML 2009; their paper compared progressive ordering with human education and animal training, while tracing the idea of starting small at least to Elman (1993).

The original hypothesis was twofold: an easy-to-hard ordering can speed up convergence and help non-convex optimization avoid poor local minima. Bengio framed it as a kind of continuation method, starting from a smoothed version of the problem and sharpening it step by step.

How difficulty is measured

The hard part is defining what counts as an «easy» example. Answers fall into two families. The first relies on fixed, hand-designed heuristics: the length of a sentence or sequence, the number of objects in an image, the noise level, or difficulty labels annotated by hand. The second lets the model decide for itself: in the self-paced learning of Kumar, Packer and Koller (2010), each iteration selects the lowest-loss examples as easy and relaxes a weight until the full dataset is in play. In reinforcement learning, the teacher-student scheme of Matiisen and colleagues favors the tasks on which the agent improves fastest, gauged by the slope of its learning curve. Controlled studies have quantified difficulty using the loss of a reference model or a consistency «c-score».

When it helps and when it doesn't

The evidence is mixed, and worth stating plainly. Wu and colleagues (ICLR 2021) showed that on standard, clean datasets a random order performs as well as or better than a curriculum, and that much of the credit usually given to ordering actually comes from growing the training set dynamically. The benefits become clear in two cases: a tight training-time budget and noisy data. With large models or easy tasks, ordering examples may add nothing and can even slow things down. The reverse order —anti-curriculum, hardest first— generally does not help in those controlled experiments, though other studies have found it competitive in specific settings; the question remains open.

Where it is used

Curriculum learning shows up mainly in three areas. In computer vision, grading image complexity; in natural language processing, moving from short, frequent text to longer constructions; and in reinforcement learning, where chaining subtasks of rising difficulty can crack problems that direct training cannot, as the teacher-student framework demonstrated on navigation tasks. The survey by Petru Soviany and colleagues, submitted in 2021, documents applications across many tasks but identifies two recurring practical limits: ranking examples reliably from easy to hard and choosing the right pace for introducing harder ones.