Attention

Attention is a mechanism that lets a neural network weigh different parts of its input dynamically, according to how relevant each part is to the element it is currently producing. Instead of squeezing an entire sequence —a sentence, a paragraph, an image— into a single fixed-size vector, the network learns to «look» at the portions of the data that actually matter at each step and to give them more weight.

This capacity for selective focus, loosely inspired by the way the human mind concentrates perception on some things while ignoring others, is now the central component of today's large language models. Transformers made it a central component of many translators and systems that draft, summarise or hold a conversation. Primary source.

What it solves: weighing the input

Early sequence-to-sequence models carried a bottleneck. An encoder read the whole source sentence and compressed it into a fixed vector; a decoder then had to reconstruct the target sentence from that vector alone. The longer the input, the more information was lost in forcing it through that narrow channel. Attention breaks the bottleneck: it keeps every representation of the input available and lets the model decide, for each word it generates, how much weight to give to each part of the source. Primary source.

From translation to self-attention

The mechanism was born in neural machine translation. In 2014, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio proposed letting the decoder softly «search» for the source words relevant to predicting each target word, rather than relying on a single vector. The conceptual leap came with self-attention: instead of one sequence attending to a different one, every element of the same sequence attends to all the others. A word can thus connect with another many positions away and settle, for instance, what a pronoun refers to. Primary source.

Query, key, value and the Transformer

Self-attention is built from three vectors per element: a query (query), a key (key) and a value (value). Each element's query is compared with every other element's key to measure their affinity; those affinities, once normalised, become weights that combine the values, yielding a representation enriched with the relevant context. To capture several kinds of relationship at once, the process runs in parallel across several «heads» (multi-head attention). In 2017, Ashish Vaswani and colleagues at Google pushed the idea to its limit in “Attention Is All You Need”: they built an entire architecture, the Transformer, on attention alone and dispensed with recurrence completely. By allowing a sequence to be processed in parallel, that choice opened the door to training far larger models.

Why it matters, and its quadratic cost

Full self-attention compares every element with every other one, so its cost grows quadratically with sequence length. This is a practical limit for long contexts. Primary source.