Transformers

A Transformer is a neural network architecture based entirely on the attention mechanism, which processes a whole sequence—a text, for instance—at once, dispensing with recurrence and convolutions. Eight Google researchers introduced it in 2017, in the paper “Attention Is All You Need,” led by Ashish Vaswani. It is the architecture on which today's large language models are built.

The attention mechanism

Its central idea is self-attention: for each element of the sequence—each “token”—the model weighs the relevance of all the others and builds its representation from that context. Multi-head attention repeats that computation in parallel with several “heads,” each attending to a different kind of relationship. Because attention alone does not know word order, the Transformer adds a positional encoding—sinusoidal in the original design—that injects each token's position.

Why it displaced recurrent networks

Recurrent networks (RNNs, LSTMs) used to dominate; they process the sequence word by word, in order. The Transformer has no recurrence, so it processes all tokens in parallel. In their paper, the authors summed it up this way: they propose an architecture “based solely on attention mechanisms, dispensing with recurrence and convolutions entirely,” more parallelizable and faster to train. That parallelization is what made it possible to train ever-larger models.

Encoder, decoder and variants

The original design has two blocks: an encoder, which reads the input, and a decoder, which generates the output. Three families derive from it. Encoder-only models, such as BERT (Google, 2018), read text bidirectionally and serve to classify, search or answer questions. Decoder-only models, such as the GPT series (OpenAI, since 2018), predict the next token and are the basis of text generation. And encoder-decoder models are used in tasks such as translation.

Its limit: quadratic cost

Standard attention has a cost that grows quadratically with sequence length: doubling the number of tokens quadruples the computation and memory. Hence an active research line to make it cheaper, with sparse or linear attention variants and more memory-efficient implementations such as FlashAttention. Alternative architectures with linear cost, such as state-space models, are also being explored; in 2026, however, the Transformer remains the reference architecture for large language models.