Multi-Task Learning

Multi-Task Learning (MTL) trains a single network to solve several related tasks at once, rather than building a separate model for each one. What makes it work is that the tasks share the same internal representation: the layers that extract features from the input are common to all of them, so what the model learns while solving one task becomes available to the others.

Rich Caruana formalized the idea in his paper «Multitask Learning» (1997), where he described it as a form of inductive transfer: learning several tasks in parallel over a shared representation lets what is learned for one task help the others be learned better.

Why it helps: inductive bias and regularization

Caruana framed MTL as an inductive bias: the information carried in the training signals of neighboring tasks pushes the model toward hypotheses that explain more than one task, which improves generalization. Sebastian Ruder, in his 2017 survey «An Overview of Multi-Task Learning in Deep Neural Networks», describes the same effect from another angle: because each task brings its own signal and its own noise, training them together implicitly increases the effective sample size and acts as regularization, reducing overfitting. More signal and sturdier representations are the payoff MTL offers when the tasks are genuinely related.

Hard versus soft sharing

There are two ways to divide up the parameters. In hard sharing, the most common approach, every task literally shares the same hidden layers and only splits at the end into a task-specific «head»; forcing a common trunk sharply cuts the risk of overfitting. In soft sharing, each task keeps its own model and its own parameters, but the distance between them is penalized so the solutions resemble one another without being identical. Hard sharing is cheaper and more rigid; soft sharing is more flexible and more expensive.

When it hurts: negative transfer

Sharing does not always add up. When tasks compete or bear no relation to each other, forcing them onto a single trunk degrades the result: this is negative transfer. On top of it sits the problem of balancing each task's loss, since a task that dominates training can drown out the rest; there is no universal recipe for weighting them, and it remains an area of active research. MTL should not be confused with transfer learning, which moves what has been learned from one task to another in sequence, one after the other; multi-task learning trains them all at the same time. That distinction has its own entry in this glossary. Today multi-task learning is a central ingredient of large models, which handle many tasks with a single network.