Statistical language models: fundamentals and applications

On 30 July 2026, the most visible language models use neural networks, but their defining question remains statistical: what probability does the model assign to the next element given its context? A model may answer with count tables, a recurrent network or a Transformer. The architecture changes; the object being estimated—a conditional distribution over sequences—remains.

This article retains the useful path from n-grams to BERT and GPT, but organises it around one skill: reading any language model through five decisions—unit, context, estimation, evaluation and use. That distinction separates a model that predicts sequences from a complete system that translates, summarises or conducts a conversation.

A sequence becomes a series of predictions

A language model assigns probability to a sequence of units. Those units may be characters, words or tokens produced by a segmenter. The chain rule decomposes the joint probability into a product of conditional probabilities: the first unit under an initial context, the second given the preceding units, and so on. To generate, a system estimates a distribution for the next token, selects or samples one, appends it to the context and repeats.

This formulation is not the same as understanding meaning. In his 1948 paper, Claude Shannon explicitly separated semantic aspects from the engineering problem of selecting messages under a distribution. A model can capture highly useful regularities in a corpus without holding a verifiable representation of what its sentences refer to.

The unit changes the problem. A word vocabulary is legible but creates many rare or unknown elements. Characters remove that closed vocabulary at the cost of longer sequences. Subwords seek a compromise. Two models trained on the same text are therefore not directly comparable when they use different tokenisers unless the evaluation accounts for that difference.

N-grams turn text into counts

An n-gram approximates the full history using only the last n − 1 units. A bigram estimates a word from one previous word; a trigram uses two. A probability can be obtained by dividing the count of a continuation by the count of its context. The model is inspectable: its counts can be queried and the fragments supporting an estimate can be identified.

Its limitation appears immediately. The number of combinations grows with vocabulary and context length. Many reasonable sequences never occur in the corpus. If the model assigns zero probability to one transition, the product for the entire sentence is zero. Accumulating more counts without a sampling strategy does not solve every issue: domains, periods and language varieties may still be absent.

Smoothing reserves probability for unseen events

Smoothing techniques discount probability mass from observed events and redistribute it among rare or absent ones. Backoff uses a shorter context when a longer one lacks evidence; interpolation combines orders. The original work by Kneser and Ney showed that the backoff distribution should not merely use a word’s overall frequency: the number of different contexts in which it appeared mattered.

This detail teaches a broader rule. When a product claims to “handle inputs it has never seen”, ask what probability mass it reserves for them and which information it uses when specific evidence runs out. An n-gram exposes the answer in counts and formulas. A neural model generalises through shared representations and learned parameters, but it still must be tested on unseen cases.

Perplexity measures prediction, not complete utility

Mean cross-entropy penalises the probability assigned to correct units in a test set. Perplexity is an exponential transformation of that quantity: lower usually means better prediction of that corpus under that tokenisation. It should not be compared without checking the unit, vocabulary, test set and handling of unknown words.

Lower perplexity does not guarantee a better chatbot, translator or summariser. It is an intrinsic measurement of the language objective. An application adds search, instructions, decoding, filters and external data; it then needs use-specific metrics and suitable human review. The language model is a probabilistic component, not the entire application.

From discrete counts to distributed representations

N-grams treat two different contexts as unrelated even when they play similar roles. In 2003, Bengio and colleagues proposed learning distributed word representations and the sequence probability function together. Nearby contexts in that space could share evidence. The paper locates the bridge precisely: this is still statistical language modelling, but estimation no longer depends only on a table of exact fragments.

Recurrent networks added a state updated while traversing a sequence. In principle it can summarise a variable history; in practice, training may lose distant signals. LSTM introduced gates and a memory path to control what is preserved, written and read. It does not “remember” like a person: it maintains numerical values because the training objective rewarded that behaviour.

Transformer changes access to context

The Transformer by Vaswani and colleagues replaced recurrence in its translation system with attention and feed-forward layers. Each position computes combinations of other positions permitted by a mask. This makes training more parallelisable than a strictly step-by-step recurrence. It does not mean every word receives equal weight or that context is infinite: attention assigns weights, the architecture retains limits, and cost grows with length according to the mechanism used.

The key lesson for reading later models is to separate architecture from objective. Transformer describes operations and connections. The pretraining objective determines which information helps reduce loss.

BERT and GPT: one family, different questions

BERT uses a Transformer encoder and trains bidirectional representations by conditioning on both left and right context. Its distinctive task hides input tokens and attempts to recover them. The original work then adapts the model with an output layer to tasks such as inference and extractive question answering. “It understands better” is not an adequate summary: the task, adaptation and benchmark must be named.

OpenAI’s first GPT paper combined generative pretraining—prediction from left to right—with discriminative fine-tuning. In a causal model, the position being predicted cannot inspect future tokens. That restriction matches generation: produce the next token from the available prefix.

“BERT is for understanding and GPT is for writing” is therefore only a rough mnemonic. A masked encoder produces representations that can be adapted to tasks; a causal model produces continuation probabilities and can also transfer. What can be verified is the mask, objective, data, adaptation procedure and results, not a human faculty attached to an acronym.

Applications: the model scores, the system decides

In speech recognition, a language model helps score candidate sequences alongside acoustic signals. In translation, a decoder distributes probability over outputs conditioned on the input. In autocomplete it ranks continuations. In an assistant, the result also depends on system instructions, document retrieval, tools, the decoding algorithm and downstream controls.

The same separation exposes risks. Corpus bias, stale information and spurious associations can alter the distribution. A probable continuation can be false. A model with a good average may fail in an underrepresented language or domain. The remedy is not to claim that the model “understands almost like a human”, but to define the use, build a representative evaluation set and preserve a path for review and correction.

A card for reading any language model

• Unit: character, word or token, and which segmenter?
• Context: which positions may be used, and what is the effective limit?
• Estimation: smoothed counts, recurrent state, attention or another parameterisation?
• Objective: next token, masked token, conditioned sequence or another loss?
• Evaluation: comparable perplexity and task metrics on unseen data?
• System: which retrieval, decoding, filters or tools alter the output?

The transferable skill is reducing any language-model claim to those six decisions. N-grams expose the logic of probability and scarcity; neural networks learn representations that share evidence; BERT and GPT show how the objective changes the use of one architectural family. The acronym identifies a mechanism. Evaluation sets the boundary of what it can actually do.