Autoencoders

An autoencoder is a neural network trained to reconstruct its own input. Rather than predicting an external label, it simply tries to copy whatever it receives back to the output, with one deliberate catch: along the way the data must squeeze through a bottleneck, a middle layer far smaller than the input. That constraint stops the network from memorizing and forces it to keep only what matters.

Because the target signal is the input itself, no human labels are needed. This is self-supervised (or unsupervised) learning, guided by a single reconstruction error, usually the mean squared difference between input and output.

Encoder, decoder and the latent code

Every autoencoder has two halves. The encoder compresses the input into a lower-dimensional representation, the so-called code or latent vector. The decoder runs the process backward, rebuilding the original input from that code. If the reconstruction is faithful despite passing through a narrow space, the code has captured the essential structure of the data. In 2006, Geoffrey Hinton and Ruslan Salakhutdinov showed in Science that deep autoencoders could learn low-dimensional codes far more useful than those from principal component analysis (PCA).

What they are good for: dimensionality reduction, denoising and anomalies

Their most classic use is dimensionality reduction: as a nonlinear relative of PCA, an autoencoder projects complex data onto a handful of variables to compress or visualize it. A well-known variant, the denoising autoencoder, is trained on deliberately corrupted inputs and learns to return the clean version; Pascal Vincent and colleagues formalized it in 2008. And because the network only reconstructs well what resembles the data it has already seen, it is a natural fit for anomaly detection: whatever is rebuilt with a large error is flagged as unusual, handy in fraud or predictive maintenance. Dimensionality reduction and anomaly detection each have their own entry in this glossary.

The generative variant: the variational autoencoder

The classic autoencoder compresses, but it is not designed to invent new data. That ability arrived with the variational autoencoder (VAE), proposed by Diederik Kingma and Max Welling in 2013. Instead of a single code, its encoder learns an entire probability distribution over the latent space; by sampling from it, the decoder produces plausible examples that were never in the original data. That is why the VAE is considered a generative model, kin to GANs, and earns its own entry in this glossary. Nearly two decades on, autoencoders remain one of the most elegant ideas in representation learning: teaching a machine to understand data by asking it, quite simply, to draw the data again.