Transformer-based language models: concepts and advances

On 30 July 2026, “uses a Transformer” describes a family of components, not a complete model or a quality guarantee. Two systems can share the label while differing in masks, position signals, head counts, encoder-decoder structure, objectives, context windows and inference cost. The useful skill is reading the diagram as a contract: which positions connect, what information they share, and what price the system pays for doing so.

The architecture published in Attention Is All You Need combined attention, feed-forward layers, residuals, normalisation and position in a translation system. Later language models select and alter those pieces. Reconstructing them prevents “attention” from becoming a magical explanation.

The block input has shape, not human meaning

After tokenisation, a sequence of length n is represented as a matrix: one row per position and d values per row. That width, the model dimension, must be preserved across residual connections. The block receives no words; it receives vectors contextualised up to that point.

Each head learns three linear projections of the matrix. Q contains queries, K keys and V values. If a head uses dimension dh, every position produces a query, key and value of that size. The names describe algebraic roles, not separate databases.

A dot product turns compatibility into mixing

Scaled attention can be summarised as softmax(QKᵀ/√dh)V. QKᵀ creates an n × n matrix: one score between every query and permitted key. Dividing by the square root of the dimension prevents large products from saturating softmax as width grows. The mask turns forbidden connections into values receiving zero weight. Softmax normalises every row; multiplication by V mixes content.

This resolves three misconceptions. Attention does not “look at every word” when a mask or sparse pattern prevents it. A high weight does not by itself prove a cause of the final prediction, because layers, residual paths and other heads remain. And the output need not copy one token: it combines learned value vectors.

Multiple heads expand projections, not facts

Multi-head attention runs the calculation in different subspaces, concatenates the outputs and projects them back to model dimension. This allows different relationships to be represented without forcing them into one compatibility matrix. But the number of heads does not measure how many types of reasoning a model possesses. A head may be redundant, specialise only on certain data or change its pattern across layers.

Auditing a claim about a head requires interventions rather than only a visualisation: ablate it, alter inputs and measure the effect on outputs. A coloured map is descriptive evidence of weights, not a complete behavioural explanation.

Position is an architectural decision

Self-attention without an extra signal is indifferent to order. The original Transformer added sinusoidal vectors to representations and also considered learned positions. Other variants incorporate relative relationships. RoPE, for example, rotates queries and keys according to position to introduce relative dependence into the attention product.

The choice affects extrapolation and cost. A model trained under one length or position distribution does not automatically preserve accuracy at longer windows. An interface accepting more tokens proves input capacity only; information retrieval and use of distant positions need targeted tests.

The complete block alternates communication and transformation

Attention lets positions exchange information. The feed-forward network transforms each position separately through two projections and a nonlinearity. Residual connections add a sub-block’s input to its output, creating direct routes through depth. Normalisation controls scale and may appear before or after sub-blocks depending on the variant.

These pieces matter during training. Gradients must cross many layers, and small changes in order, initialisation or normalisation affect stability. Comparing models only by parameter count therefore hides decisions that alter usable capacity.

Encoder, decoder and encoder-decoder are not synonyms

An encoder normally permits bidirectional attention within the input and produces one representation per position. BERT is the classic example: it hides tokens during pretraining and uses context on both sides to recover them.

A causal decoder applies a triangular mask: the position predicting a token can use only the prefix. The first GPT used left-to-right generative pretraining and task adaptation. During generation, every new token joins the prefix and triggers the next calculation.

An encoder-decoder separates input from output. The encoder processes the source; the decoder uses causal self-attention over generated content and cross-attention over source representations. The original Transformer had this form. T5 studied it within a text-to-text framework. The right family depends on whether the task requires representing an input, continuing a prefix or generating an output conditioned on another sequence.

Parallel training, sequential generation

During causal training, a complete sequence allows losses at many positions to be computed together because the mask blocks the future. During generation, the next token is unknown and steps are sequential. Without optimisation, each step would repeat calculations over the prefix.

A KV cache stores keys and values from earlier positions for reuse. It reduces repeated computation, but cache memory grows with length, layers, head size and the number of K/V heads. Grouped-query attention (GQA) shares fewer key and value heads among multiple query heads, seeking a trade-off between quality and inference cost. The advance is not “more intelligence”; it is another allocation of memory and bandwidth.

Long context has more than one cost

A dense attention matrix contains position pairs, so its size grows quadratically with n. This affects operations and intermediate memory. FlashAttention does not change the mathematical output of exact attention: it tiles the calculation to reduce reads and writes between GPU memory levels. Separating algorithm from architecture prevents the claim that every time reduction changes model capability.

Other methods change the pattern. Longformer combines local windows with global-attention positions to reduce growth relative to dense attention. The cost is that not every pair connects directly. Before accepting “longer context”, ask whether the technique is exact, sparse, recurrent or compressive and which tasks were evaluated.

Cases should demonstrate mechanisms, not decorate them

A Transformer can operate on text, images, audio or other sequences when an appropriate tokenisation or representation is defined. That does not turn a CNN, protein model and assistant into the same product. The unit, objective and evaluation change. A “case studies” section without an entity, number, source and control merely imitates the appearance of evidence.

An honest case links the primary work and names the transferred component. BERT demonstrates a masked encoder on specific tasks; GPT a pretrained causal decoder; T5 a text-to-text encoder-decoder formulation; FlashAttention an IO-aware exact implementation. None proves universal Transformer superiority over every alternative.

The architecture reading card

• Representation: length n, dimension d and tokenisation.
• Attention: number and size of Q, K and V heads.
• Connectivity: causal, bidirectional, local or cross-attention mask.
• Position: absolute, relative, rotary or another method, plus training length.
• Block: order of attention, feed-forward, residuals and normalisation.
• Family: encoder, decoder or encoder-decoder.
• Cost: training, KV cache, intermediate memory, latency and length.
• Evidence: task, baseline, ablation and unseen data.

The transferable skill is turning a Transformer diagram into that card. Attention defines mixing routes; the mask defines permitted information; position preserves order; the block transforms; the family sets calculation direction. An “advance” worthy of the name must say which bottleneck it changes and which cost it moves elsewhere.