Multi-Domain Learning

Multi-domain learning trains a single model to solve the same task across several domains at once, where each domain is a different data distribution. The classic example is sentiment analysis applied simultaneously to book, film and electronics reviews: the task is the same, but the vocabulary and statistics of each type of review change. Its challenge is domain shift: a model trained on one distribution degrades when applied to another.

What sets it apart from neighboring concepts

A common confusion with three nearby notions is worth undoing. Multi-task learning learns several different tasks at once, not several domains of the same task. Transfer learning (and domain adaptation) is directional: it carries knowledge from a source domain to a target one, and it cares about performance on the target; multi-domain learning, by contrast, treats all domains equally and seeks to perform well on all at once. And continual learning faces a sequence of tasks over time, with the problem of forgetting.

Where EWC fits

That distinction matters for placing a technique often cited out of place. Elastic weight consolidation (EWC, Kirkpatrick et al., 2017) fights catastrophic forgetting by penalizing changes to the weights important for earlier tasks. It belongs to continual learning, which sees tasks sequentially, not to multi-domain learning, which trains them all at once and therefore does not suffer that forgetting. In a multi-domain entry, EWC should be mentioned only to draw the distinction.

Methods

The methods proper to multi-domain learning combine a shared part across domains with parameters specific to each: a common feature extractor that aligns the distributions, domain-dependent normalization layers, and adapters—small modules inserted into a shared network and switched according to the domain, with very few parameters of their own.