Transfer Learning and Semi-Supervised Learning in Language Models

Labelling data costs money and human time. That constraint, not a shortage of ideas, has shaped much of the recent progress in language models: nearly every technique that dominates today exists to exploit the text nobody has annotated, which is practically all of it.

Two families solve that problem by different routes, and it is worth knowing which is which, because they are often conflated.

Semi-supervised learning: making use of the unlabelled

Semi-supervised learning applies when there is little labelled data and a great deal of unlabelled data. Combining both, the model learns more generalizable representations than it would from the annotated portion alone.

Self-training, or pseudo-labelling

The most intuitive method works like this: a first model is trained on the few labelled examples available; that model then labels the unannotated data; the predictions it is most confident about are added to the training set, and the cycle repeats. Each pass enlarges the data and refines the model.

It sounds like a trick that should not work — the model learns from its own answers — and yet it does, on one condition. The work that documents it best is «Self-training with Noisy Student» (Xie, Luong, Hovy and Le, 2019): they pseudo-labelled 300 million unlabelled images and reached 88.4% top-1 accuracy on ImageNet, two points above the previous state of the art, which required 3.5 billion weakly labelled Instagram images.

The condition is the interesting part, and it names the method: the student is injected with noise — dropout, stochastic depth, data augmentation — while the teacher labels without it. Learning the same answer under harder conditions than those in which it was generated is what makes the student generalize better than its teacher. Without that noise, the loop merely amplifies the initial errors.

A note of honesty about the genealogy: the original formulation of pseudo-labelling is attributed to Dong-Hyun Lee (2013), but it was a workshop paper with no archived proceedings and has no verifiable official host. That is why it is not linked here: the attribution is given, and the reader is pointed to the later work that is primary and checkable.

Contrastive learning

The other route does not try to guess labels but to do without them. Contrastive learning trains the model to pull together the representations of two versions of the same example and push them away from those of different examples. Nobody says what anything is; only what resembles what.

Its reference demonstration is SimCLR (Chen, Kornblith, Norouzi and Hinton, 2020): a linear classifier trained on representations learned this way reaches 76.5% top-1 on ImageNet, matching a supervised ResNet-50. And fine-tuned with just 1% of the labels it reaches 85.8% top-5. That is: nearly all the performance, with a hundredth of the annotation work.

When supervision comes from text that already exists

There is a third route combining the previous two, worth knowing because it explains much of what works today: using as a label something that was already written alongside the data.

The reference case is CLIP (OpenAI, 2021), trained on 400 million image-text pairs gathered from the internet. Nobody annotated those images: the caption was already there. The model learns to pull each image towards its description and push it away from the rest — contrastive learning, applied across two different kinds of data.

Its result, verbatim: «we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples». That is, without having seen that particular test's training set at all.

It is worth understanding what this means and what it does not. «Without labels» does not mean without supervision: it means the supervision was supplied by millions of people writing captions, and the model got it for free. The annotation did not disappear; it changed hands. And the biases of whoever wrote those texts travel with it — the counterpart that rarely accompanies the headline.

And there is a practical consequence worth keeping to hand when evaluating any system trained this way: the quality of the result depends on the quality of text nobody wrote with training a model in mind. A caption can be ironic, wrong, or describing something else entirely. At a scale of hundreds of millions that averages out, but it does not vanish — and it explains why these models fail in ways that look capricious. Asking what corpus supplied the supervision is, in practice, asking where the errors will come from.

Transfer learning: reusing what was learned

Transfer is a different thing, though it chases the same saving. Here there are two models: a source, pre-trained on a task with abundant data, and a target, fine-tuned to a specific task where data is scarce. Knowledge travels from the first to the second.

The practical detail is that this handover can be spoiled. If the whole model is fine-tuned at once with a high learning rate, the general knowledge is overwritten: the phenomenon known as catastrophic forgetting. Hence the two usual precautions — freezing layers and fine-tuning gradually.

The case that pinned it down: ULMFiT

In 2018, Jeremy Howard and Sebastian Ruder published «Universal Language Model Fine-tuning for Text Classification», which turned those precautions into a reproducible method. Their figures, verbatim from the abstract: it reduces the error by 18-24% on the majority of datasets and, «with only 100 labeled examples, it matches the performance of training from scratch on 100x more data».

That second figure is worth pausing on, because it is the entire argument for the approach: a hundred examples against ten thousand. For any organization that cannot afford a team of annotators, it is the difference between being able to use the technology and not.

And the one that generalized it: BERT

That same year, BERT took the idea to bidirectional pre-training and set a new state of the art on eleven tasks, pushing the GLUE score to 80.5% — a 7.7-point absolute improvement — and SQuAD v1.1 F1 to 93.2.

The connection is worth seeing: BERT is pre-trained by solving tasks that need no human annotation — predicting hidden words in any text — and only afterwards fine-tuned with labelled data. That is semi-supervision and transfer working together, which is how they are almost always used in practice.

What remains unsolved

Adaptation to new domains is still fragile: a model pre-trained on general text can perform poorly on medical or legal text, where vocabulary and structure change. And there is a problem that is not technical but a matter of responsibility: if the pre-training corpus carries biases, transfer propagates them to every derived task, including those nobody had in mind when it was built.

The capability: three questions for a model trained «with little data»

1. Little labelled data, or little data? Not the same thing. Nearly all these techniques need enormous amounts of unannotated text; what they save is the human work of labelling, not the volume.

2. What was it pre-trained on? Performance on your task depends on how closely your domain resembles the pre-training one. A model trained on general text and evaluated on general text says little about what it will do with your contracts.

3. Is the figure you are shown from pre-training or from fine-tuning? They are distinct phases and are measured differently. Confusing them is the commonest error in reading these announcements.

The deep end

• Howard and Ruder (2018), ULMFiT, with the detail of the three gradual fine-tuning techniques.
• Devlin and others (2018), BERT.
• Xie and others (2019), Noisy Student, to understand why noise is the key to self-training.
• Chen and others (2020), SimCLR, for contrastive learning explained from its components.
• Radford and others (2021), CLIP, to see how text already on the internet becomes supervision.