Language Models in Machine Translation and Language Assistance Systems

Machine translation is probably the artificial intelligence application most people use without thinking about it. It is also one of the few with a public, annual, independent evaluation anyone can consult — which makes it the best place to learn how to read a quality claim, here and almost everywhere else.

The progression, with the figures for each leap

The starting point is statistical models: unigrams, bigrams and trigrams built on Markov chains, estimating the probability of a word from the one, two or three before it. They work, and their limit is obvious: a three-word window cannot see the sentence.

The first neural leap came in 2014 with «Sequence to Sequence Learning with Neural Networks», by Ilya Sutskever, Oriol Vinyals and Quoc Le. An LSTM reached 34.8 BLEU on the full WMT'14 English-to-French test set, against 33.3 for the reference phrase-based statistical system. And when they used the LSTM to rescore the thousand hypotheses that system produced, they rose to 36.5.

That same year, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio published «Neural Machine Translation by Jointly Learning to Align and Translate», which diagnosed the bottleneck of the earlier approach — compressing an entire sentence into a fixed-length vector — and proposed letting the model search for the relevant parts of the source at each step. The attention mechanism is born there. It should be said that its abstract gives no BLEU figure: it claims performance «comparable» to the best phrase-based system of the day.

Consolidation came in 2017 with «Attention Is All You Need», which dispensed with recurrence entirely: 28.4 BLEU on WMT 2014 English-German, more than two points above the best previous result including ensembles, and 41.8 BLEU on English-French with a single model, after «3.5 days on eight GPUs». That last figure matters as much as the first: the Transformer did not only translate better, but far more cheaply.

The figure you have to know how to read

In 2016 Google presented its production neural system in a technical paper with a claim that has been repeated thousands of times: it reduces translation errors «by an average of 60%».

It is true, and it is almost always told wrongly. The 60% comes from a comparative human evaluation against Google's own phrase-based statistical system — not against a human translator — and measured on simple, isolated sentences, not running text. With those two qualifications in place it remains an enormous advance. Without them, the sentence suggests something the paper does not say.

This is exactly the kind of slippage worth learning to spot, because it recurs in every announcement of every technology: a real improvement, measured against a specific baseline, presented as though it were absolute.

Where it stands today: the place where you check

Here is what almost no article on machine translation mentions, and it is the most useful part: there is an annual conference, WMT, that puts systems through human evaluation with professional annotators and publishes the results openly.

The 2024 results carry a title that says it: «the LLM era is here but MT is not solved yet». They evaluated eleven language pairs with 28 participants, plus eight large language models and four commercial providers. The finding that matters: human references are still in the winning cluster in 7 of the 11 pairs. That is, in most cases no system conclusively beat translation done by people.

Two qualifications from the same report, both instructive. First: English-Spanish was the easiest pair, and there the best systems produced near-flawless translations — so a Spanish speaker's experience with these tools is not representative of how they work in general. Second: the automatic metrics misled. One system came first in the automatic ranking and did not win the human evaluation.

The 2025 edition rose to 30 language pairs and 60 systems, and its title is a methodological warning aimed at the field itself: it is time to stop evaluating on easy test sets.

Why the easy tests stopped working

The 2025 edition carries a warning to the field in its very title: it is time to stop evaluating on easy test sets. It is worth understanding what that means, because it is a general problem of measurement and not only of translation.

As systems improve, an exam that once separated the good from the bad stops separating anything: everyone passes and the differences fall inside the margin of error. The test did not get easier; the candidates got better, and the effect is the same — the instrument stops discriminating.

WMT's answer in 2025 was to sample by difficulty and seek harder texts, widening to 30 language pairs and 60 systems evaluated, with human judgment on half the pairs across four or five domains. That is: they did not measure more, they measured harder.

And there lies the consequence that serves you as a reader. When a system announces a sky-high score on a veteran benchmark, that figure says as much about the test as about the system. The useful question is not what it scored but when the exam was designed and whether it still tells anyone apart. Ninety-nine per cent on a test almost everyone passes is not a feat: it is an exhausted test.

Language assistance: the other half

The same technology underpins grammar and style checkers and the writing tools that suggest continuations. The difference from translation is the success criterion: in translation there is an original to check against, whereas in language assistance what is offered is a preference. Which is why «accuracy» figures in this second family deserve suspicion: they often measure agreement with a style guide, not correctness.

The capability: three questions for any «translates better than»

1. Measured on what text? Isolated simple sentences are not running text. Machine translation errors concentrate precisely in what demands long memory: pronouns, cross-references, terminology that must stay consistent across a document.

2. Compared with what? «60% fewer errors» requires knowing fewer than what. Almost always the baseline is the previous version of the same product, not a professional translator.

3. Automatic or human evaluation? A score like BLEU compares against a reference translation and is useful for fast iteration, but WMT documents that it can rank systems wrongly. When the decision matters, what counts is the judgment of professional annotators.

The deep end

• Sutskever, Vinyals and Le (2014), the first neural leap, with its three BLEU figures.
• Bahdanau, Cho and Bengio (2014), the origin of the attention mechanism.
• Vaswani and others (2017), the Transformer.
• Wu and others (2016), Google's system, to read the 60% figure in its context.
• WMT 2024 and WMT 2025, the open annual evaluations. They are long, but their results sections read without technical training and are the only way to check for yourself which system leads today.