Recurrent Neural Networks

A recurrent neural network (RNN) is an architecture built to process sequences —text, speech, time series— one element at a time rather than all at once. Its defining feature is the hidden state: a memory vector that the network updates at every step and that summarizes everything seen so far.

This state is fed back into the network alongside the next input, so the output at any moment depends not only on the current item but on the whole accumulated context. That internal memory is what lets an RNN capture the order and the temporal dependencies of a sequence —something a conventional, feedback-free network cannot do.

How they work: hidden state and backpropagation through time

At each step, the RNN combines the current input with the previous hidden state using the same weights, reused again and again across the sequence. With a fixed number of parameters it can therefore handle sequences of any length. Training relies on backpropagation through time (BPTT), formalized by Paul Werbos in 1990: the network is «unrolled» through time as if it were a deep network with shared weights, and the error is propagated backward across every step of the sequence.

The gradient problem and the LSTM and GRU solutions

That training runs into a deep obstacle. Bengio, Simard and Frasconi showed in 1994 that learning long-range dependencies with gradient descent is difficult: as the error is propagated many steps back, the gradient tends to vanish —shrinking toward zero, so the network forgets distant information— or to explode, growing out of control. Exploding gradients are curbed by clipping their norm; vanishing gradients demanded a redesign of the cell itself.

The most influential answer was the LSTM (Long Short-Term Memory), introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997: it adds a memory cell and a set of gates —input, forget and output— that decide what to store, discard and emit, keeping a stable flow of error across many steps. In 2014, Kyunghyun Cho and colleagues proposed the GRU (Gated Recurrent Unit), which streamlines the idea with just two gates —update and reset—, fewer parameters and often comparable results.

RNNs versus Transformers today

In 2017, the paper «Attention Is All You Need», by Vaswani and his team, introduced the Transformer, an architecture built on the attention mechanism that processes the entire sequence in parallel and dispenses with recurrence altogether. Because it does not carry information forward step by step, it trains far faster and relates distant elements more easily. It has since largely displaced RNNs in language processing and underpins today's large language models.

The handover, though, is not complete. Attention costs grow quadratically with sequence length and it needs to see the full input, whereas an RNN advances one step at a time with a compact state. That keeps it relevant for streaming processing, for resource-constrained devices and for certain time-series tasks. How to combine recurrent efficiency with the quality of attention remains an open problem: recent lines of work, such as state-space models, are exploring exactly that frontier.