Context Is Not Memory: Where a Language Model Stores Information

As of July 30, 2026, many products promise that a language model “remembers.” That word may describe at least five incompatible mechanisms: learned parameters, tokens still in view, recurrent state, retrieved documents, or data stored by the application. Attention is not a sixth store; it is an operation for combining available information. Drawing these layers makes it possible to diagnose why a system forgets, when a conversation becomes persistent data, and which memory claim is merely marketing.

First: attention selects; it does not archive

In the original Transformer, every position constructs a query, keys, and values. Matches between the query and keys weight a mixture of values. That operation transforms representations already present in the sequence; by itself, it creates no archive that survives another request.

The mask determines which positions are visible. An encoder may combine context on both sides; an autoregressive generator is prevented from seeing future tokens. Multiple heads apply different projections, but “paying attention” does not mean consciously deciding what to remember. It is computation conditioned on inputs and weights.

The simplest test is to start a session with no history. If the fact disappears, it lived in context or temporary application state, not in the parameters. If it returns because the service injects a summary, a storage and retrieval system exists around the model. Calling all of this memory hides the very mechanism that must be evaluated.

Layer 1: parameters retain regularities

During training, gradients change weights to reduce prediction error. Those weights condense statistical patterns from many examples. They may sometimes reproduce a fact, style, or even a sequence, but they do not contain source-and-date rows that can be directly queried. Locating, correcting, or deleting a particular datum is not equivalent to editing a database.

This “parametric memory” persists across requests as long as the same model version is used. It is also frozen relative to the training cutoff unless further training occurs. An answer emerging from weights is not proven true, and its provenance is not automatically known. Apparent knowledge must therefore be distinguished from verifiable retrieval.

Layer 2: the context window is a workbench

Tokens in a request and its supplied history form the material available to the current computation. The system can relate an early instruction to a later question because both remain in that window. Once the limit is exceeded, the application must truncate, summarize, or select; each decision can lose details.

Nominal capacity is not perfect use. Lost in the Middle placed relevant information at different locations in long contexts and observed pronounced variation in retrieval performance. A window should be tested by moving evidence, adding distractors, and asking questions that connect fragments, not merely by showing that the text fits.

Context is not necessarily private or ephemeral, either. A provider or application may log prompts for operations or product purposes. That belongs to service policy and architecture, not to model attention. An audit must ask what is sent, who stores it, for how long, and under which controls.

Layer 3: recurrent state crosses steps or segments

A recurrent network updates a state while moving through a sequence. The LSTM architecture added gates and a cell to control what information enters, remains, and leaves. The state is a learned compression of the past, not a literal copy; details can decay, and it is normally reset at the start of another sequence.

Transformer-XL addressed dependence beyond fixed segments by reusing hidden states from a previous segment and adopting relative positional encoding. That is segment-level recurrence. It should not be confused with sparse attention, where a pattern limits which position pairs connect. Longformer is an example of local and global sparse-attention patterns for long sequences.

There is also a key-value cache during generation. It prevents the states of earlier tokens from being recomputed at every step and speeds inference. Despite the word cache, it adds no new facts and does not independently extend the model’s trained limit; it preserves intermediate results from the same sequence.

Layer 4: retrieval from an external store

A system can search documents, a vector database, or an index and place selected passages in the prompt. Retrieval-Augmented Generation combined parametric memory with indexed non-parametric memory and evaluated it on knowledge-intensive tasks. RETRO studied another large-scale integration of retrieval into generation.

The operational advantage is clear: a corpus can be updated, each passage can retain metadata, and readers can receive a citation. But retrieval does not verify. Search may choose the wrong document, a passage may lose context, and the generator may claim more than the source says. The chain should record query, index version, results, used passages, and final claims.

Similarity search is not human recall, either. It maps the query and documents into representations, retrieves neighbors under a metric, and may rerank candidates. Quality depends on chunking, embeddings, filters, language, and coverage. If information is absent from the corpus or not retrieved, the model cannot “remember” it through this route.

Layer 5: the product retains durable state

Preferences, tasks, names, or decisions may be stored in a conventional database. In the next conversation, the application selects some records and injects them again. This layer has properties a model alone does not: fields, identity, timestamps, permissions, change history, and explicit deletion.

Early Memory Networks already separated components for writing, reading, and answering over a memory. Modern product systems may implement the principle with very different technology. The decisive question is not whether a component is named “memory,” but who writes what, under which key, how it is retrieved, and how a user corrects or deletes it.

Persistence and updating are different axes

A datum can persist for a long time and be difficult to update, like an association in model weights. It can be short-lived and exact, like a figure in the current prompt. It can persist and remain editable, like a database record. It can be current but poorly retrieved from an index. One memory score blends properties that should be measured separately.

Record at least six: scope, which sessions share it; duration, when it expires; capacity, how much fits; fidelity, what is lost; provenance, whether its origin is known; and control, who can read, correct, and delete it. Latency and cost complete the technical comparison.

A memory test must include interference

Asking immediately about a freshly supplied sentence is not enough. Introduce similar information, correct a fact, change the subject, fill the window, close the session, and return. Then check whether the system retrieves the right version, cites its origin, merges identities, or honors an instruction to forget.

For external retrieval, measure recall of relevant documents before grading the prose. For durable state, test permissions and deletion. For long context, vary position and distractors. For parameters, do not promise point updates or provenance that the mechanism cannot provide.

The map for any “memory” claim

For a new feature, draw five boxes: weights, context, state or cache, retrieval, and database. Mark where the datum originates, how long it lasts, how it reaches the model, and who can change it. Then identify attention’s role: selecting relationships within visible material, not guarding it until tomorrow.

That map resolves the central confusion. A model can attend to a sentence without keeping it; retain patterns without citing them; retrieve a document without verifying it; and appear to remember because an application reintroduces a note. “Memory” becomes a defensible property only when the store, route, and test are named.