DistilBERT

DistilBERT is a compact version of BERT obtained through knowledge distillation. Victor Sanh and colleagues at Hugging Face presented it in 2019. Per its original paper, it reduces BERT's size by about 40% (from 110 to 66 million parameters), is roughly 60% faster at inference, and retains about 97% of its performance on the GLUE benchmark.

How distillation works

Knowledge distillation, formalized by Hinton, Vinyals and Dean in 2015, trains a small model (the “student”) to reproduce the output probability distribution of a large model (the “teacher,” here BERT), rather than learning only from the correct labels. Those softened probabilities convey how the teacher generalizes—which classes it deems similar—a richer signal than the hard label. DistilBERT combines three losses during pre-training: the distillation loss, BERT's own masked-language-modeling loss, and a similarity loss between the two models' internal states. Its architecture halves the number of layers (from twelve to six) and is initialized from every other layer of the teacher.

An important nuance

It is worth being precise about what distillation does, because it is often described poorly. DistilBERT does not remove “redundant parameters” from BERT nor prune it. A new, smaller network is designed and trained to imitate the large model's behavior. The size reduction comes from an architectural decision—fewer layers—and the retained performance from exploiting the teacher's generalization signal. This differs from pruning, which does remove weights from an already trained model, and from quantization, which lowers the numerical precision of the weights.

Where it stands today

DistilBERT dates from 2019, the era of BERT-style (encoder) models, and was an influential demonstration that a compact model can approach the large one at far lower cost—valuable for deploying in production with low latency. It remains a common starting point for classification or information-extraction tasks. It is not, however, the state of the art in compression: today's toolkit combines quantization, pruning, low-rank adapters such as LoRA, and a new generation of small language models, where distillation is one tool among several rather than the only one.