The Future of Language Models in Artificial Intelligence

Every few months a bigger model appears and the headline is always the same: more parameters, more capability. It is worth knowing which part of that equation is documented and which is not, because the technical papers themselves are far more cautious than the announcements that cite them.

The architecture that made it possible

The basis of everything current is the transformer, which replaced recurrent architectures. Its centrepiece is the attention mechanism: instead of processing text word by word in order, the model weighs the relevance of all parts of the input at once. That removes sequential processing and allows massive parallelism, which is what makes training at this scale viable.

The paper that introduced it, «Attention Is All You Need» (2017), gave two figures worth reading together: 28.4 BLEU on WMT 2014 English-German translation, more than two points above the best previous result including ensembles, and 41.8 BLEU on English-French with a single model, after «3.5 days on eight GPUs».

That second half — the cost — is the one almost never quoted, and it is what explains why the whole field moved to this architecture. It did not win on accuracy alone: it won because it was far cheaper to train.

Pre-train and fine-tune: why it works

The dominant method has two phases. First self-supervised pre-training: the model learns from enormous amounts of unlabelled text by solving tasks that require no human annotation, such as predicting hidden words. Then fine-tuning with labelled data for the specific task.

BERT (2018) took that idea to bidirectional context: understanding a word in relation to those before and after it, not in one direction only. Its results: state of the art on eleven tasks, with the GLUE score at 80.5% — a 7.7-point absolute improvement — and SQuAD v1.1 F1 at 93.2.

Scale, and what its own paper admits

The other route was to grow. GPT-3 (2020) was trained with 175 billion parameters, «10x more than any previous non-sparse language model», and its novelty was being applied without gradient updates or fine-tuning: the task and a few examples are given directly in the input text.

And here is the detail worth retaining, because it separates a scientific paper from a brochure: the authors themselves declare the datasets on which that few-shot learning fails. It is not an awkward footnote; it is part of the result. When someone cites GPT-3 as proof that scale solves generalization, that section of the paper is the answer.

The cost, and how it is being contained

Models this size raise a practical problem of compute and energy. The most widespread answer is knowledge distillation: training a small model to reproduce the behaviour of a large one. DistilBERT (2019) is the reference case: it reduces model size by 40% while retaining 97% of its language-understanding capabilities and being 60% faster.

Those three figures should be read for what they are: three separate measurements. The 97% refers to a standardized test suite, not to the specific task of whoever deploys it.

The cost attention cannot dodge

One property of the attention mechanism is worth understanding because it decides what can be deployed and what cannot. Comparing every word with every other means the work grows with the square of the text's length: doubling the input does not double the computation, it quadruples it.

This is not an engineering footnote. It is why context windows are announced as feats, and why processing one long document costs disproportionately more than processing ten short ones. It also explains why alternatives persist: in 2019, a Facebook AI team showed in «Pay Less Attention with Lightweight and Dynamic Convolutions» that dynamic convolutions reached 29.7 BLEU on WMT'14 English-German translation — the state of the art at the time — at a cost that scales linearly with length.

The lesson for a reader is not that one architecture beats another, but that every gain in quality is paid for in compute, and that price decides what reaches the product. When an excellent model never shows up in the tool you actually use, the explanation usually lies here rather than in the laboratory.

Bias: a data problem, not an architecture problem

Size aggravates a problem it does not create. If the pre-training corpus contains biases — and every corpus scraped from the internet does — the model learns them along with the grammar, and subsequent fine-tuning propagates them to every derived task. This is a matter of data provenance and evaluation, not something fixed by changing architecture.

The other future: smaller and open

The dominant story is that models grow. The missing half is that, in parallel, there is a race to make them smaller and to publish them with open weights — and there distillation stopped being a compression trick and became a training method.

The Gemma 2 technical report (Google DeepMind, 2024) says it plainly: they trained «the 2B and 9B models with knowledge distillation instead of next token prediction», while the 27-billion model followed the conventional route. Within one family, some siblings are distilled and others are not.

DeepSeek-R1 took the idea further in 2025, deriving «six dense models (1.5B, 7B, 8B, 14B, 32B, 70B) distilled» from a larger one on the Qwen and Llama families, and releasing them openly. The work ended up in Nature.

For anyone watching from outside, the practical consequence is what matters: the model that ends up inside your phone or your company will not be the biggest, but the best of those that fit. And that frontier moves by distillation and quantization, not by scale.

It is also worth not conflating the two routes. Quantization stores each parameter in fewer bits and does not change how many there are; distillation trains a model with fewer parameters to imitate a larger one. They are announced together and measured differently, and only the second genuinely changes the model's size. A claim that mixes them is describing two savings as though they were one.

The capability: what to ask of the next large model

1. Are the parameters the news, or the result? A parameter count describes size, not what the model can do. It is the easiest figure to publish and the least informative on its own.

2. What does the paper say about where it fails? Serious work declares it — GPT-3 does. If the material you are shown contains only wins, you are not reading the paper: you are reading the press summary.

3. What did it cost, and what does it cost to use? Training and inference are different costs, and the second is the one you will pay if you deploy it. A model that does not fit your compute budget is not an option, however good it is.

The deep end

• Vaswani and others (2017), the transformer paper, with quality and cost figures together.
• Devlin and others (2018), BERT, for bidirectional pre-training.
• Brown and others (2020), GPT-3. Worth seeking out the sections where the authors describe their own limitations.
• Sanh and others (2019), DistilBERT, to understand what is kept and what is lost in compression.
• Wu and others (2019), the head-to-head between convolution and self-attention, with the linear-versus-quadratic cost argument.