Conditional Generative Models

A conditional generative model is one that learns to generate data conditioned on additional information—a condition or label—modeling the distribution p(x|y) instead of the general distribution p(x). That condition can be a class, a text, another image or a map, and it serves to control what is generated, rather than sampling blindly.

Conditional versus unconditional

An unconditional generative model produces some plausible example of the domain, but does not allow asking which; it samples from p(x) without control. The conditional one adds the condition to the input of the generative process and thereby directs the output: instead of any digit, it generates “a 7”; instead of an arbitrary image, it generates an image “from a text.” In exchange, it requires well-prepared data and condition signals.

How it is implemented

The idea has been applied to the main families of generative models. The conditional GAN (Mirza and Osindero, 2014) feeds the condition to both the generator and the discriminator; from it derives pix2pix, for translating one image into another. The conditional variational autoencoder does the same with that type of model. And diffusion models conditioned on text—the basis of generating images from a description—use guidance techniques to balance the image's quality and its fidelity to the instruction.

What it is used for

Conditional generative models are behind text-to-image generation, image-to-image translation (colorizing a photo, turning a sketch into an image), speech synthesis and targeted data augmentation, which generates examples of a specific class to balance a dataset.