Discriminative Models

A discriminative model is one that learns to distinguish classes by directly modeling the probability of the label given the input, p(y|x), or a decision boundary that separates the classes. It does not model how the data are generated; it only learns to separate them.

Discriminative versus generative

The key distinction is with the generative model, which models the joint distribution p(x,y)—that is, how the data of each class are distributed. A generative model can therefore generate new examples and also classify by applying Bayes' theorem; a discriminative model, by contrast, does not learn a full data distribution for sampling through that objective. In a sentence: the discriminative one learns the boundary between classes; the generative one learns what the data of each class look like. Primary source.

Not the same as a “classifier”

A common confusion is worth undoing: “discriminative” and “generative” are not synonyms for “classifier.” A classifier is any model that assigns labels, and it can be built with either approach. Logistic regression, support vector machines, decision trees and classification neural networks are discriminative; the naive Bayes classifier and linear discriminant analysis are generative. The difference is not whether they classify, but what they model. Primary source.

Which one suits, and the current landscape

A classic paper by Andrew Ng and Michael Jordan (2001) showed there is no universal winner: the discriminative approach tends to reach a lower error when there is plenty of data, while the generative one converges faster with little data, because its stronger assumptions act as a form of regularization. Today, most general-purpose classifiers are discriminative, while the large generative models—language and diffusion models—model the distribution of the data so as to sample and generate new content.