Restricted Boltzmann Machines

A Restricted Boltzmann Machine (RBM) is a two-layer generative stochastic neural network: a visible layer that represents the data and a hidden layer that learns latent features. The word «restricted» is the defining detail: units connect across the two layers but never to other units in the same layer. That absence of intra-layer links is precisely what makes the model tractable to train — something its unrestricted ancestor, the general Boltzmann machine, never was in practice.

Unlike a network that computes a deterministic output, an RBM learns a probability distribution over the data. It does not answer «this is a cat»; it models how plausible each possible configuration is, and it can generate new examples that resemble the ones it has seen.

Two layers and an energy function

RBMs belong to the family of energy-based models. Every joint configuration of visible and hidden units is assigned a scalar number, its energy, determined by the connection weights and the bias of each unit. The probability of a configuration follows from that energy through the Boltzmann distribution: the lower the energy, the more likely the configuration. Training an RBM means adjusting weights and biases so that configurations resembling real data have low energy — and therefore high probability — while the rest have high energy. Because units within a layer are not connected to one another, the units of one layer are conditionally independent given the state of the other, which lets them be sampled in parallel in a single sweep.

How they are trained: contrastive divergence

The training obstacle is that computing the distribution's normalizing term exactly is intractable. The solution that made RBMs popular was contrastive divergence, an approximate algorithm that Geoffrey Hinton set out in «Training Products of Experts by Minimizing Contrastive Divergence» (Neural Computation, 2002). Rather than running a sampling chain all the way to equilibrium, contrastive divergence takes only a few steps of Gibbs sampling — often just one — and updates the weights from that approximate gradient estimate. It is fast, works surprisingly well, and was the piece that turned RBMs into a practical tool.

Their historical role and status today

The architecture dates back to 1986, when Paul Smolensky introduced it under the name «Harmonium». Its moment, though, came in 2006: Hinton, Simon Osindero and Yee-Whye Teh showed in «A Fast Learning Algorithm for Deep Belief Nets» that stacking RBMs and pre-training them one layer at a time, without labels, gave deep networks a good starting point. Those deep belief networks revived interest in deep learning at a time when training many-layer networks looked hopeless. RBMs also shone in the Netflix Prize: Salakhutdinov, Mnih and Hinton applied them to collaborative filtering in 2007.

Their current standing deserves honesty. RBMs are today largely out of use. Layer-by-layer pre-training stopped being necessary once end-to-end training, with more data and GPUs, proved good enough on its own. And as generative models they have been displaced by variational autoencoders, generative adversarial networks (GANs) and, more recently, diffusion models. Their interest now is mainly historical and pedagogical: they are one of the most elegant demonstrations of energy-based generative modeling and an essential chapter for understanding where deep learning came from.