Knowledge Distillation: How to Read a Small Model's Promise

If your phone fixes a sentence, summarizes a notification or transcribes a voice note without sending anything to a server, it is running a model that fits in your pocket. That model was not born small: it is almost always the reduced version of a much larger one. The procedure has a name — knowledge distillation — and an open primary literature worth knowing, because that is where the numbers in the advertisements come from.

Let us take those numbers, and then what they actually mean.

The idea, and why it was born in 2006

The approach predates the current fashion. In 2006, Cristian Buciluă, Rich Caruana and Alexandru Niculescu-Mizil, of Cornell University, presented a paper at the KDD conference titled «Model Compression». They proposed «a method for compressing large, complex ensembles into smaller, faster models, usually without significant loss in performance».

Their motivation was not theoretical. The best systems of the day were ensembles: hundreds or thousands of classifiers voting together. Excellent, and completely unusable anywhere memory and compute are scarce. The examples in the paper are of their time and startlingly current: handheld organizers and hearing aids.

Almost twenty years later, that is exactly why there are language models inside telephones.

The mechanism: teaching with doubts, not with answers

The formulation everyone uses today arrived in 2015, with «Distilling the Knowledge in a Neural Network», by Geoffrey Hinton, Oriol Vinyals and Jeff Dean. Their proposal: use «the class probabilities produced by the cumbersome model as soft targets for training the small model».

It is worth pausing here, because this is the only part you really need to grasp. When a system classifies a handwritten image, it does not merely say «this is a 2». It emits a distribution: very likely a 2, somewhat likely a 7, almost impossible a 5. That gradation is information. It says twos resemble sevens more than they resemble fives, and the large model learned that from millions of examples. The bare correct label throws that information away.

Distillation means training the small model against that whole distribution instead of against the flat answer. And so the small probabilities become visible, they are «softened» with a parameter called temperature: raising it, the paper says, «produces a softer probability distribution over classes».

The experiment proving it sits in the same text and is easy to follow. On MNIST, the handwritten digit set: the large network makes 67 test errors; a small network trained conventionally makes 146; that same small network, trained only on soft targets at temperature 20, drops to 74. It recovers nearly the whole gap without changing size.

The second experiment is even more eloquent. In speech recognition, an ensemble of ten models reached 61.1% frame accuracy and a 10.7% word error rate; a single model distilled from that ensemble reached 60.8% and the same 10.7% word error rate. Ten working models compressed into one, without losing the metric that mattered.

Now the «97%»: what it measures and what it does not

In October 2019, a team at Hugging Face published DistilBERT, with the sentence that defined the genre: it reduces the model's size «by 40%, while retaining 97% of its language understanding capabilities and being 60% faster».

All three figures are true and sit in the paper's abstract. But they are three measurements of three different things, and reading them in one breath leaves an impression none of them supports:

The 40% is model size. The 60% is speed. And the 97% is performance on a set of language-understanding tests: a standardized exam, not the specific task of whoever will use it. Nothing guarantees that 97% holds in a particular domain — contracts, medical records, customer support in one industry — because nobody measured it there. The claim is honest; the extrapolation is not the paper's.

Where this stands today

The technique stopped being a laboratory trick some time ago.

The Gemma 2 technical report, published by Google DeepMind on 31 July 2024, states that they trained «the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction». The 27-billion-parameter model, by contrast, was trained conventionally. Within one family, some siblings are distilled and others are not.

DeepSeek did the same in January 2025 with R1, from which it derived «six dense models (1.5B, 7B, 8B, 14B, 32B, 70B) distilled from DeepSeek-R1 based on Qwen and Llama». The work was eventually published in Nature (645, 633-638, 2025).

Apple's case deserves its own paragraph, because it teaches document reading. Its June 2024 announcement describes an on-device model of «~3 billion parameters», but says nothing about distillation: what it details is quantization, averaging 3.7 bits per weight. Distillation is documented a year later, in the 2025 technical report, which states that they retrained the dense on-device model over the last 10% of tokens «using a distillation loss from the MoE teacher».

Two different techniques, two different documents, a year apart. Attributing the first to the wrong document is a small error that reveals a reading that never happened.

What gets inherited along with the knowledge

One consequence follows directly from the mechanism and almost never appears in announcements. If the small model is trained to reproduce the large one's probability distribution, then it also reproduces whatever the large one gets wrong. The teacher's judgment and the teacher's biases travel down the same channel, because they are the same channel.

This does not invalidate the technique; it places it. A distilled model is not a cleaned-up version of its teacher. It is a cheaper version. Asking who the teacher was is therefore a technical question, not a curiosity.

The capability: three questions for any small model

1. Smaller in which unit? Having fewer parameters is not the same as storing each parameter in fewer bits. Apple's case illustrates it: 3.7 bits per weight cuts memory without touching the parameter count. If the announcement omits the unit, the number means nothing.

2. That performance percentage, measured on which exam? Every «retains X%» refers to one specific test suite. Ask for its name. And remember your task was not in it.

3. Who was the teacher? A distilled model inherits the judgment of the model it learned from, failures included. If the teacher is not declared, half the spec sheet is missing.

The deep end, with a lesson about reaching sources

The three foundational papers here are within anyone's reach, but not all through the same door:

• Hinton, Vinyals and Dean (2015) and Sanh, Debut, Chaumond and Wolf (2019) are on arXiv, open and complete.
• Buciluă, Caruana and Niculescu-Mizil (2006) sits behind the ACM paywall. But one of its authors keeps a public copy on his Cornell page, something many publishing contracts expressly permit the author to do.

That distinction is worth learning: a copy the author hosts at their own institution is not the same as a copy uploaded by a third party. The first is self-archiving, an established and legitimate practice; the second is not. When a paper looks inaccessible, the author's own page or their university repository is the first place to look, and the right one.

The capability you leave with: faced with any model boasting of being small, separate the three numbers — size, speed, performance — demand the exam the third was measured on, and ask which model it learned from. Those questions worked in 2006 and will keep working when the names change.