From N-Grams to Transformers: How Language Models Evolve
A history of language models read through five axes: representation, objective, context, adaptation, and evaluation.
Re-edited on July 30, 2026, this account begins with a distinction: the history of natural language processing is not exactly the history of language models. Symbolic systems, grammars, and computational dictionaries were essential, but a language model has a more specific function: it assigns probabilities to sequences or estimates which unit may occur given a context.
The evolution is not a clean march from “primitive rules” to “understanding” either. Each stage changes a constraint: how a word is represented, how much context can be used, which objective is optimized, how work is shared across tasks, and how a system is adapted. Reading those axes reveals why a new model improves and what its score does not demonstrate.
From counting sequences to measuring uncertainty
In A Mathematical Theory of Communication, published in two parts in 1948, Claude Shannon treated language as a statistical source and showed successive approximations to English obtained by considering progressively longer dependencies. His goal was a theory of communication, not a conversational assistant, but he established two lasting ideas: a sequence can be modeled through conditional probabilities, and its uncertainty can be quantified.
An n-gram model approximates the probability of the next unit by looking only at a short window. Such models are straightforward to count and query, but the number of combinations grows quickly. An unseen sequence receives zero probability without smoothing; a rare word has little evidence; two similar contexts share no information unless the designer provides a mechanism. Even so, n-grams established a pattern that continues: train by predicting units of text, then evaluate how surprising the model finds a held-out sample.
Perplexity summarizes that average surprise: lower perplexity means the model assigned higher probability to the test text. It is useful for comparing models on the same tokenization and dataset, but it is not a universal measure of utility. Changing the vocabulary changes the number; better probability does not guarantee factuality, safety, or quality on a downstream task.
Networks learn shared representations
The neural transition was not merely a matter of using a larger machine. The 2003 paper A Neural Probabilistic Language Model learned a distributed representation for each word together with the probability function for sequences. Words occurring in similar contexts could share features in a continuous space. That property addresses sparse counts: the model can generalize to combinations it did not observe exactly.
Bengio’s model still used a fixed-length context. Recurrent neural networks instead update a state while traversing a sequence, allowing computation to depend on a variable history. A paper by Mikolov and colleagues at Interspeech 2010 compared a recurrent model with a backoff language model on two speech tasks and reported improvements in word error rate. The result applied to defined conditions; it was not evidence of general understanding.
Recurrence also imposes costs: later steps depend on earlier ones, limiting parallelism, and long dependencies are difficult to learn. Gated variants such as LSTMs and GRUs helped retain information, but did not remove the sequential bottleneck. The central question for the next stage was how to relate distant positions without forcing all information through a single state updated step by step.
Computed context: from ELMo to the Transformer
A word does not have one representation in every use. “Bank” changes between a financial institution and the side of a river. ELMo, presented in 2018, built representations from the internal states of a deep bidirectional language model; those representations depended on the full sentence and could be incorporated into different task models. Context became part of the vector for each occurrence, rather than only part of later computation.
A year earlier, Attention Is All You Need had introduced the Transformer for translation. Its core architecture replaced recurrence and convolutions with attention: each position computes which other positions should influence its representation. That allowed training to be parallelized more effectively and connected distant tokens more directly. The paper neither invented attention nor demonstrated every later use, but it provided an architecture that could scale and be reused.
“Attention” does not mean that a system knows which part is true or important to a person. It is a learned operation that combines representations. Attention weights are not automatically a causal explanation for an output. To understand what the architecture contributes, examine the objective, data, context length, tests, and interventions instead of turning the mechanism’s name into a human faculty.
Pretrain once, adapt in different ways
BERT demonstrated in 2018 a bidirectional pretraining strategy based primarily on recovering masked tokens. The full model was then fine-tuned with labeled data for tasks including classification, inference, and extractive question answering. It was not a conversational generator in its original design: its objective and use differed from autoregressive models that predict the next token.
T5 systematically studied transfer learning in 2020 and expressed tasks as text input to text output. Translation, summarization, and classification could share a format, parameters, and pretraining, although each task still needed its own definition and evaluation. “Text-to-text” is a learning interface; it does not make the errors of translation and classification equivalent.
Also in 2020, GPT‑3 evaluated a 175-billion-parameter autoregressive model using instructions and examples written in its context, without updating gradients for each test. The paper documented zero-, one-, and few-shot results across many tasks, but it also reported limitations, including uneven performance and methodological concerns about data contamination. In-context learning changed the adaptation interface; it did not guarantee that the model acquired a stable rule or that its output was true.
How to read a model’s genealogy
A useful historical record starts with five columns. Representation: discrete symbols, static vectors, or context-conditioned vectors. Objective: next token, masked token, sequence-to-sequence, or another criterion. Context: fixed window, recurrent state, or attention with a stated limit. Adaptation: rules, labeled fine-tuning, full tuning, additional modules, or examples in context. Evaluation: perplexity, downstream task, human review, and conditions of use.
That table prevents calling every later publication a “successor.” BERT and GPT‑3 share the Transformer but not the same objective or direction of prediction. A model with more parameters may perform worse when data, training budget, or tests change. A complete system may also add document retrieval, tools, filters, and rules: crediting every outcome to the language model erases decisive components.
When comparing two generations, the axis has to be held constant. The same test should preserve examples, instructions, scoring criteria, and allowed resources; otherwise it is impossible to tell whether the model improved or the exam changed. Training cost should be separated from response cost, and the exact model version should be recorded. A score without those details is difficult to reproduce, while a demonstration selected by a manufacturer can hide adverse cases. Results should include failures, not only a best-case sample. Technical history becomes auditable when each advance is expressed as a measurable difference under declared conditions.
It also requires unsupported claims to be removed in practice. The original article presented a GPT‑3 medical chatbot as a notable case of accurate, personalized information, but named no study, system, protocol, or result; that assertion is not retained. The transferable skill is reconstructing any “new generation” by asking what changed in representation, objective, context, adaptation, and evaluation. Chronology informs; those five axes explain.
This article was produced with artificial intelligence under human editorial oversight.
