Attention Mechanisms

An attention mechanism lets a neural network dynamically weigh which parts of the input matter most for producing each part of the output, instead of squeezing an entire sequence into a single fixed-size vector. Rather than treating every element equally, the model learns to assign a weight to each one and to concentrate its computation where the information counts.

That ability to «look» selectively removed a structural limitation of earlier seq2seq architectures and is now the centrepiece of large language models.

The origin: Bahdanau's machine translation (2014)

Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio introduced attention in 2014, in «Neural Machine Translation by Jointly Learning to Align and Translate». Neural translators of the day followed the encoder-decoder scheme: one network compressed the whole source sentence into a fixed-length vector and another expanded it into the translation. That bottleneck degraded quality on long sentences, because a single vector cannot retain all the information. Their fix let the decoder consult every encoder state and compute, at each step, a weighted combination of them: as it generates each word, the model «aligns» and attends selectively to the relevant source words. This variant is known as additive or Bahdanau attention, and the recurrent networks that hosted it were typically LSTMs (see LSTM).

The query-key-value scheme and self-attention

The formulation that prevailed casts attention as three roles: a query, a set of keys and a set of values. The model measures the similarity between the query and each key to obtain the attention weights, normalised to sum to one, and uses them to average the values. The result is a reading of the content skewed toward what is relevant to that query. When queries, keys and values come from the same sequence, we call it self-attention: each element relates to every other element in the sequence, capturing dependencies between distant words without stepping through the sentence one token at a time. This direct all-pairs computation is also the root of its main limitation, discussed below.

The Transformer and multi-head attention

In 2017, Ashish Vaswani and colleagues at Google pushed the idea to its limit in «Attention Is All You Need»: their architecture, the Transformer, dispenses with recurrence entirely and builds all processing on attention (see Transformer). Two contributions define it. Scaled dot-product attention computes the weights from dot products between queries and keys, divided by the square root of their dimension to stabilise training. And multi-head attention runs several attention operations in parallel, each with its own query, key and value projections, so that every «head» learns to focus on a different kind of relationship. By removing recurrence, all elements are processed at once, which made training far more parallelisable. Attention is now the core of large language models.

Open questions remain. The cost of self-attention grows quadratically with sequence length —comparing every element with every other— which makes long texts expensive to process and has spurred many efficient variants. And the interpretability of the weights is still debated: a head attending strongly to a word is not, by itself, an explanation of why the model decides what it decides.