GRU

The GRU (Gated Recurrent Unit) is a type of unit for recurrent neural networks that uses a mechanism of “gates” to control what information to keep and what to discard as it processes a sequence. That control helps the network capture long-range dependencies and mitigate the vanishing gradient problem that afflicted simple recurrent networks.

Where it comes from

It was introduced by Kyunghyun Cho and his collaborators in 2014, in a work on machine translation with an encoder-decoder architecture. Its name, “gated recurrent unit,” was cemented that same year in a later study that compared it with other units.

How it works

The GRU has two gates. The update gate decides how much information from the past to keep and how much to replace with the new. The reset gate decides how much of the previous state to ignore when computing the candidate information. With those two gates, the unit regulates its own short- and long-term memory.

Versus the LSTM

The GRU is, in essence, a simplified version of the LSTM (Long Short-Term Memory, by Hochreiter and Schmidhuber, 1997). The LSTM has three gates and a separate memory cell state; the GRU makes do with two gates and without that separate cell state, which gives it fewer parameters and, often, faster training. As for performance, the two are comparable and which wins depends on the task: there is no universal victor.

Its place today

It is worth placing it honestly. For years, GRUs and LSTMs were the standard tool for modeling sequences—time series, language processing, speech recognition. But since the arrival of transformers in 2017, these recurrent architectures have been largely displaced in language processing, giving ground to a model that processes the sequence in parallel and better captures long dependencies. The GRU retains value where simplicity and efficiency matter.