An introduction to language models in artificial intelligence

On 30 July 2026, a modern language model can receive a sentence and produce a fluent continuation within seconds. Describing it as a machine that “understands words” provides nothing testable. A useful explanation follows the computation: text → tokens → vectors → Transformer layers → logits → probabilities → selection. Each arrow contains a decision that affects what the model can see, learn and return.

A language model assigns probabilities to sequences. That definition, which also covers n-grams, is the right entry point. The modern difference lies in how context is represented and the distribution estimated. The reader’s skill is to reconstruct one token prediction without confusing the base model with the conversational interface around it.

First step: text becomes tokens

The network does not receive printed words. A tokenizer converts characters into identifiers from a finite vocabulary. A token may be a word, fragment, punctuation mark or smaller unit depending on the system. The SentencePiece paper, for example, describes a subword tokenizer that can be trained on raw text without prior language-dependent segmentation.

Tokenisation determines length and coverage. A frequent word may occupy one token while another is split into several. The same sentence can have a different cost and effective context limit under two tokenizers. Tokenisation also affects which regularities are easy to share across inflections, scripts or languages. “Accepts 100,000 tokens” does not mean the same number of words, documents or amount of information for every input.

Special markers for starts, ends, separators or roles are often added before the first ordinary token. In a conversation, the system may serialise instructions and messages in a format seen during fine-tuning. That protocol belongs to the input; it does not arise spontaneously inside the weights.

Second step: every identifier becomes a vector

An identifier looks up a row in an embedding table. The result is a vector of numbers adjusted during training. Tokens used in contexts that demand similar behaviour can acquire representations that share evidence. The neural probabilistic model by Bengio and colleagues made this bridge between distributed representations and sequence probability explicit.

An embedding is not a dictionary definition. It is an initial state for computation, dependent on vocabulary, data and objective. As it traverses the layers, the representation at a position incorporates information from the permitted context.

Because attention alone does not know order, the architecture adds positional information. The original Transformer used sinusoidal encodings and also considered learned positions. Later models use other mechanisms. Without a position signal, two sequences containing the same tokens in a different order would be difficult to distinguish through pure attention.

Third step: Q, K and V organise attention

In one self-attention head, each representation is projected into three vectors: query (Q), key (K) and value (V). For a given position, the product of its query with the keys produces compatibility scores. These are scaled, a mask is applied, and softmax turns them into weights. The output is a weighted sum of values.

A search metaphor can help: Q expresses which signal the position needs; K describes which signal each position offers; V contains the material mixed when it receives weight. But these are not readable sentences or rules. They are learned projections whose role depends on the entire network.

Multi-head attention repeats the process with different parameters and concatenates the results. A head need not correspond to a stable linguistic category. Some may display interpretable patterns on examples; this does not prove that they learned a universal rule or that the model reasons through a list of human-like modules.

The mask decides which information is allowed

A causal mask prevents the position used to predict the next token from inspecting future positions. During training, many positions in a sequence can be processed in parallel, but every prediction retains that restriction. During generation, tokens do appear one at a time because the next input depends on what has just been produced.

A bidirectional encoder can permit a position to attend on both sides. BERT hides some input tokens and trains the model to recover them from left and right context. The difference from a causal model is not that one “understands” and the other “writes”: they use different visibility patterns and loss objectives.

A mask can also block padding or separate segments. When a system uses local attention, sliding windows or compressed memory, “all context is available” may be false even when the input nominally fits. The important question is which positions are actually connected.

A layer also transforms each position

The attention output passes through residual connections, normalisation and a feed-forward network applied to each position. Residual connections preserve a route for the previous representation while adding transformations. Normalisation controls internal scales. The feed-forward network expands and projects the vector again through nonlinear operations.

Stacking layers repeats communication among positions and local transformation. “Depth” means more parameterised stages, not guaranteed understanding. A model can build useful representations while also exploiting spurious correlations or failing when inputs move to another domain.

From representation to logits

At the end, a projection produces one number for every token in the vocabulary. These numbers are logits: unnormalised scores. Softmax converts them into a distribution summing to one. The highest-probability token is only one possible output. The system may sample, restrict candidates, change temperature or apply penalties.

Temperature changes the shape of the distribution before sampling. A lower temperature concentrates mass and makes output more stable; a higher one disperses it. Top-k retains a fixed number of candidates; top-p retains the smallest set reaching a selected cumulative probability. These rules do not correct facts or add knowledge: they change how a choice is made from model probabilities.

A conversational answer adds further external layers: message templates, instructions, document retrieval, tools, filters and perhaps another reviewing model. Attributing all behaviour to “the language model” hides the system that actually produced the response.

How it learns without human labels for every token

In causal pretraining, the text supplies its own targets: given a prefix, the observed next token acts as the label. In masked training, part of the input is hidden and the original is the target. This is commonly called self-supervised learning because the signal is constructed from the data, rather than conventional supervised learning in which a person manually labels every case.

The loss compares the predicted distribution with the target token. Backpropagation computes gradients and the optimiser adjusts parameters. The process does not store an explicit copy of every rule; it compresses regularities useful for lowering loss under a data and compute budget.

The first GPT paper demonstrated generative pretraining followed by task adaptation. T5 studied a framework that expresses language tasks as text-to-text and compares objectives, data and transfer methods. These works do not make every problem identical: they show that a pretrained representation can be adapted when input, output, data and evaluation are specified.

Evaluation: separate prediction, task and product

Loss or perplexity on unseen text measures the language objective under one tokenisation. A question-answering benchmark measures another behaviour under a format. A product test measures whether a person completes a task better at acceptable cost and risk. No level substitutes for the next.

Length, language, domain, data date and decoding method should also be recorded. An aggregate result can conceal that a system loses accuracy on long documents, changes under another template or performs worse for a particular language variety.

The minimum trace for a claim

1. Tokenisation: how is the input split and what does it cost?
2. Representation: which embeddings and position signal are used?
3. Connectivity: which positions does the mask allow the model to see?
4. Computation: how do attention, residual and feed-forward operations combine?
5. Output: which logits are transformed and how are they decoded?
6. Objective: causal, masked, conditioned or fine-tuned with another signal?
7. Evaluation: which part measures the base model and which the complete system?

The transferable skill is reconstructing that trace before accepting words such as “understanding” or “reasoning”. A modern model does not receive words or return certainties: it receives tokens, transforms vectors and distributes probability. Utility appears when that mechanism is integrated into a system and passes a test corresponding to real use.