Boltzmann Machines

A Boltzmann machine is a type of recurrent, stochastic, generative neural network based on energy: its units are binary and activate probabilistically, and the model defines an energy over its configurations and, from it, a probability distribution. Geoffrey Hinton and Terry Sejnowski proposed it in the mid-1980s, and its name refers to the Boltzmann distribution of statistical physics.

Structure

It has two groups of units: the visible ones, which represent the data, and the hidden ones, which capture latent regularities, with symmetric connections between them. The general Boltzmann machine, with all units connected, is very hard to train: the cost grows unmanageably with its size.

The practical version: the RBM

The useful variant is the restricted Boltzmann machine (RBM), which limits the connections to a bipartite graph—visible with hidden, no connections within each group—which makes training tractable via a method called contrastive divergence. Around 2006-2012, stacking these machines built deep belief networks, which served to pre-train neural networks; they were also used in recommender systems.

A historical milestone, now in disuse

They deserve an honest placement: Boltzmann machines were highly influential—they introduced energy-based models into deep learning and helped kick-start its rise—but they are now largely in disuse for most tasks. They have been displaced by other generative architectures, such as variational autoencoders, GANs or diffusion models, and by the direct training of deep networks, which made that pre-training unnecessary. They retain value as a conceptual milestone.