Representation Learning

Representation learning (or feature learning) is the set of techniques that let a system automatically discover, from raw data, the representations or features useful for solving a task, instead of an expert designing them by hand. It is one of the pillars of deep learning.

The core idea

The intuition is that a model's success depends on how the data are represented. The same information, expressed in a different space, can make a task trivial or impossible. Representation learning seeks precisely that good representation: one in which the factors of variation underlying the data are “disentangled” and on view, so that classifying, clustering or predicting afterward becomes easier.

Versus manual engineering

Traditionally, a domain expert designed the features by hand—a laborious task limited by their intuition. Deep learning changed the paradigm: it learns hierarchies of representations directly from the data. In a vision network, the first layers detect edges; the next, textures and shapes; the deep ones, whole objects. Each level is built from the previous one.

Its forms

It takes many shapes. The layers of a deep network are, in themselves, learned representations. Embeddings turn words into dense vectors—like word2vec (2013) or GloVe (2014)—in which semantic relationships emerge as geometry. Autoencoders learn to compress and reconstruct. And today self-supervised and contrastive learning is central, learning representations from unlabeled data. The classic reference in the field is the review by Yoshua Bengio, Aaron Courville and Pascal Vincent (2013).

Why it matters

Its advantages explain much of the success of modern AI: better representations give better generalization, allow transferring what is learned from one task to another and save costly manual work. It is, in fact, the foundation of foundation models, which are trained on massive data to learn reusable general representations.