Recurrent networks for language sequence modeling

On August 4, 2013, Alex Graves submitted a paper built around a simple, productive idea: a recurrent network could generate text by predicting one unit at a time and feeding its own prediction into the next step. His paper on sequence generation with LSTM worked with words, characters and bytes. The crucial point was not that the machine stored a sentence verbatim, but that its state changed with every symbol. That distinction explains both the power and the limit of these networks: recurrence does not mean unlimited memory.

The loop that turns order into information

A conventional neural network receives an input and produces an output. A recurrent neural network, or RNN, adds a temporal connection: at step t, it combines the current input with the hidden state from step t-1. The result is a new state, passed to the following step, and an output. The same weights are reused across the sequence. That is why “dog bites man” and “man bites dog” do not follow the same computation even though they contain the same words.

The hidden state is not a file with one slot for every word. It is a fixed-size representation that compresses whatever the network has found useful in the prefix it has read. If the model processes “The doctor put the report in the drawer because she would need it later,” the state after “it” must retain enough evidence to constrain what may follow. It need not reconstruct every previous letter, but it must preserve dependencies that reduce uncertainty about the next unit.

This architecture supports several relationships between input and output. It can label every position, such as assigning a grammatical category to each word; compress a whole sequence into one output; or produce another sequence. In a causal language model, the specific task is more restrained: estimate a probability distribution for the next word, character or symbol from the history represented up to that point.

What a recurrent language model actually learns

Text is first converted into discrete units. In a word model, each position identifies a word in the vocabulary; in a character model, it identifies a letter, mark or space. The output commonly passes through a softmax function that assigns a probability to every possible unit. During training, the loss penalizes the model when it gives little probability to the symbol that really came next.

The probability of a sequence is therefore decomposed into a chain of conditional predictions: the probability of each unit depends on all earlier ones only through the state that has reached that step. That final clause matters. Two RNNs with the same vocabulary can have very different effective contexts when the state size, recurrent cell, training horizon or policy for resetting memory between sequences changes.

Graves tested that mechanism at two scales. On Penn Treebank he compared word and character prediction over a corpus of slightly more than one million words, with 930,000 for training, 74,000 for validation and 82,000 for testing. On the Hutter Prize dataset he trained on the first 96 million bytes of Wikipedia and reserved four million for validation; the state could continue for as many as 10,000 characters before a reset. Those details, documented in the same original study, matter more than saying that the network “understands context”: they define the unit, split and available horizon.

Why gradients vanish or explode

Learning requires assigning a later error to earlier decisions. In an RNN, the temporal loop is unrolled as if it were a deep network with one copy per step, then trained with backpropagation through time. The correction signal crosses the same transformations repeatedly. If those multiplications shrink its magnitude, the gradient vanishes and distant steps receive almost no credit or blame. If they enlarge it, the gradient explodes and updates can become unstable.

The analysis by Pascanu, Mikolov and Bengio examines both effects and proposes gradient norm clipping to contain explosions. Clipping does not create memory; it prevents an enormous update from derailing training. Nor is the mere existence of a state vector across hundreds of steps enough. A dependency becomes useful only when training can change the weights so that the relevant information is kept and retrieved when it affects a prediction.

Many systems also use truncated backpropagation: the state keeps moving forward, but the gradient travels back only a limited number of steps. This reduces cost and permits more frequent updates, while separating two horizons that should be reported. The model may receive old information in its state even though it learned with a much shorter credit-assignment window. “It processes 10,000 symbols” and “it learns 10,000-symbol dependencies” are not equivalent claims.

LSTM and GRU: gates that decide what to keep

Long Short-Term Memory changes the recurrent block by adding a cell and learned gates. An input gate regulates which new information enters; a forget gate determines how much of the cell state continues; and an output gate controls which part is exposed as the hidden state. The cell’s additive path makes it easier for some information and its gradient to cross more steps than in a basic RNN. It does not make memory infinite or guarantee that the network will choose the right clue.

The evidence lies in the evaluation. If a word near the beginning determines a verb form much later, performance should be measured by the distance between them rather than hidden inside a single average. The reset policy must also be stated. A network that carries state between documents may look more capable while contaminating a test with information that would not be available in real use.

A GRU applies the same broad principle with a different organization. The RNN Encoder-Decoder by Cho and colleagues introduced units with update and reset gates and learned a fixed-length representation to condition an output sequence. LSTM and GRU are not two labels for “more context”; they are different mechanisms for controlling temporal flow. They should be compared with the same data, tokenization, budget and protocol, not by the fame of the acronym.

Generating is not the same as predicting with the answer in view

During training, the model commonly receives the real previous symbol and learns to predict the next one. During generation, it receives a sample from its own distribution. An unlikely choice then becomes part of the history and can pull later predictions into a region that appeared rarely in the data. Graves described this loop of sampling, feeding back and predicting again; he also warned that short memory leaves fewer ways to recover from earlier errors.

A generation evaluation should therefore separate two tests. The first measures loss or perplexity on real text, where the history is correct. The second lets the model continue on its own and observes stability, repetition, paired punctuation, topic changes and impossible sequences. A good score on the first test does not by itself show that a long generated passage will retain structure. The input conditions are different.

What attention changed

Recurrence requires the current step to be computed after the previous one, limiting parallelism during training. The Transformer introduced in 2017 removed recurrence and convolutions from its central architecture and used attention to relate positions. This made it easier to process many positions in parallel and shortened the computational path between distant elements inside the attended window.

The comparison does not end with “attention wins.” An RNN can consume a stream one item at a time while carrying a fixed-size state, which can be useful when data arrives live or memory is constrained. Direct attention can consult representations of earlier positions, but its cost depends on how many positions are retained and compared. The right choice follows from the task: latency, length, memory, available parallelism and the kind of dependency that must survive.

How to test a claim about recurrent memory

For any RNN demonstration, the first question is which unit it predicts: word, character, byte or another segmentation. The second is where the state begins and ends. The third is how far the gradient travels during training. Without those answers, “long context” may refer to a data window, persistent memory or merely a sample that looks coherent.

The next step is to build a test that requires the past. Group examples by the distance between clue and answer, insert a more recent distracting clue and compare the model with a baseline that can see only a short window. For free generation, separately measure what happens after the first error. For streaming use, time the system symbol by symbol and record the memory occupied by its state. Each measurement targets a specific architectural property.

The minimum record includes vocabulary, handling of unknown units, sequence length, cell type and size, reset rule, gradient truncation and evaluation condition. It should also say whether a metric comes from prediction with a real history or from feedback-driven generation. Changing any one of those elements can change the result even though the label “RNN” remains the same.

The skill that remains

The durable lesson is not memorizing that LSTM came before the Transformer. It is learning to translate “the network remembers” into a testable claim: what state it carries, for how many steps, under what training procedure and in which test it demonstrably uses a distant clue. A recurrent architecture provides a path for information to move forward; effective context is whatever actually survives, changes the output and withstands a test designed to require it.