LSTM

An LSTM (Long Short-Term Memory) is a type of recurrent neural network (RNN) built to learn long-range dependencies in sequences: relationships between elements far apart in time, such as the first and last words of a paragraph. Sepp Hochreiter and Jürgen Schmidhuber introduced it in 1997.

The vanishing gradient problem

A plain RNN reads a sequence step by step, carrying an internal state that summarizes what it has seen so far. In principle that lets it remember old information, but in practice it runs into the vanishing gradient problem: during training, the error signal that should adjust the network's connections shrinks exponentially as it travels back through time. The network learns nearby relationships well but forgets distant ones. Hochreiter and Schmidhuber designed the LSTM to keep a constant error flow across many steps, so that a relevant piece of information can survive hundreds of time steps without fading away.

The cell and its three gates

At the heart of an LSTM is the cell state, a memory that runs along the sequence almost unchanged except where three gates intervene. These gates are small subnetworks that regulate the flow of information. The input gate decides what new information to store in the cell; the forget gate decides what to discard; and the output gate determines which part of the state is exposed to the rest of the network at each step. The cell should not be mistaken for a fourth gate: it is the memory store, while the gates only write to it, erase from it, or read from it. The forget gate, now considered essential, was absent from the original design; Felix Gers, Schmidhuber, and Fred Cummins added it in 2000 so the network could learn to reset itself on long, continuous streams of data.

Rise and replacement by transformers

With these gates, LSTMs dominated sequence modeling for years: machine translation, speech recognition, text generation, and time-series forecasting all relied on them. Lighter variants appeared, such as the GRU proposed by Cho and colleagues in 2014, which merges gates to simplify the computation. The turning point came in 2017 with the Transformer and its attention mechanism: rather than walking through a sequence step by step, it processes every element in parallel, which speeds up training and captures long-range dependencies more effectively. Transformers have since displaced LSTMs across most natural language processing tasks. Even so, the LSTM remains key to understanding how neural networks learned to remember, and it still proves useful in resource-constrained settings or with very long sequences where attention becomes expensive. Documentation: LSTM author's archive; original Transformer paper.