Incremental Learning

Incremental learning is a model's ability to learn from a continuous stream of data arriving in sequence, taking in new knowledge without retraining from scratch and, as far as possible, without destroying what it already knew. It stands opposite to batch learning, where the model sees a fixed, stationary dataset all at once.

Van de Ven and colleagues define it as incrementally learning from a non-stationary stream of data: the distribution shifts over time, and each new phase should add to prior knowledge rather than overwrite it. That ideal clashes with how neural networks actually learn, which is where the field's central problem begins.

Incremental, continual and lifelong: how they relate

In the literature, «incremental learning», «continual learning» and «lifelong learning» are used almost interchangeably: all three name the same goal of learning sequentially without forgetting. The differences are of emphasis, not definition. «Continual» stresses the uninterrupted stream; «lifelong» evokes a long horizon of accumulation across a system's working life; «incremental» highlights that each step adds something. The reference paper by van de Ven, Tuytelaars and Tolias is titled «Three types of incremental learning» yet frames them as continual-learning scenarios. Treating the three as distinct concepts is a mistake: they are labels for one problem.

Catastrophic forgetting: mitigated, not eliminated

The defining obstacle is catastrophic forgetting, described by McCloskey and Cohen in 1989 and reviewed by French in 1999: when a neural network trains on a new task, it tends to rapidly and drastically lose what it knew about earlier ones, because each gradient update overwrites the weights that encoded prior knowledge. Precision matters: no strategy eliminates it. At best it is mitigated, and how much depends on the scenario and on the memory and compute budget. The root cause is the stability–plasticity dilemma: a model plastic enough to learn new things is, for that very reason, prone to overwrite old ones, and reducing forgetting usually costs learning capacity.

Strategies and their real limits

Three families dominate. Regularization penalizes changes to weights deemed important for earlier tasks; its flagship is Elastic Weight Consolidation (EWC, Kirkpatrick et al., 2017), which uses Fisher information to decide which weights to protect, though its penalties degrade as many tasks pile up. Replay, or rehearsal, retrains on a sample of past data, stored or generated; it is effective but demands memory and collides with privacy. Dynamic architectures, such as Progressive Neural Networks, add modules for each new task; they avoid interference at the cost of unbounded growth and of needing to know which task an input belongs to. None solves the problem in general.

Evaluation scenarios

Comparing methods requires fixing the scenario. Van de Ven and Tolias distinguish three. In task-incremental learning the model learns separate tasks and always knows, at test time, which task it faces. In domain-incremental learning the problem's structure stays the same but the input distribution changes, with no task identity given. In class-incremental learning, the hardest, the model must recognize new classes and infer on its own which one each input belongs to, discriminating between categories it never saw together. As of 2026, per the survey by Wang and colleagues, continual learning in large language models remains an open problem: forgetting persists and is fought with replay and parameter-efficient tuning, with no definitive fix.