Attention

Attention is a mechanism that lets a neural network weigh different parts of its input dynamically, according to how relevant each part is to the element it is currently producing. Instead of squeezing an entire sequence —a sentence, a paragraph, an image— into a single fixed-size vector, the network learns to «look» at the portions of the data that actually matter at each step and to give them more weight.

This capacity for selective focus, loosely inspired by the way the human mind concentrates perception on some things while ignoring others, is now the central component of today's large language models. Without it there would be no modern machine translation, and none of the systems that draft, summarise or hold a conversation.

What it solves: weighing the input

Early sequence-to-sequence models carried a bottleneck. An encoder read the whole source sentence and compressed it into a fixed vector; a decoder then had to reconstruct the target sentence from that vector alone. The longer the input, the more information was lost in forcing it through that narrow channel. Attention breaks the bottleneck: it keeps every representation of the input available and lets the model decide, for each word it generates, how much weight to give to each part of the source.

From translation to self-attention

The mechanism was born in neural machine translation. In 2014, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio proposed letting the decoder softly «search» for the source words relevant to predicting each target word, rather than relying on a single vector. The conceptual leap came with self-attention: instead of one sequence attending to a different one, every element of the same sequence attends to all the others. A word can thus connect with another many positions away and settle, for instance, what a pronoun refers to.

Query, key, value and the Transformer

Self-attention is built from three vectors per element: a query (query), a key (key) and a value (value). Each element's query is compared with every other element's key to measure their affinity; those affinities, once normalised, become weights that combine the values, yielding a representation enriched with the relevant context. To capture several kinds of relationship at once, the process runs in parallel across several «heads» (multi-head attention). In 2017, Ashish Vaswani and colleagues at Google pushed the idea to its limit in «Attention Is All You Need»: they built an entire architecture, the Transformer, on attention alone and dispensed with recurrence completely. By allowing a sequence to be processed in parallel, that choice opened the door to training far larger models.

Why it matters, and its quadratic cost

Attention is the foundation of virtually every large language model in use today, and it has spread to computer vision, audio and biology. Yet it carries an unresolved limitation: because each element is compared with all the others, the cost grows quadratically with the length of the sequence. Doubling the input text quadruples the computation and memory that attention consumes, which makes long contexts expensive. Several research lines aim to lower it: efficient or sparse attention —Longformer, BigBird, Reformer— computes only a subset of the comparisons, and state-space models such as Mamba replace the attention matrix with a state update that scales linearly. None has yet displaced the Transformer, and cutting that cost without sacrificing quality remains an open problem.