A Map of Generative Families: VAE, GAN, Flow, Autoregressive & Diffusion

"Five of us were asked to draw the same distribution. The flow demanded an exactly reversible road. The autoregressive one wrote a left-to-right diary. The GAN hired a forger and a critic. The VAE squeezed everything through a keyhole. The diffusion model just kept erasing noise until a picture confessed. We all arrived. We took wildly different routes."

A Diffusion Model, Halfway Through Denoising

In Section 30.1 we established that modeling $p(\mathbf{x})$ means describing where natural images live and how probability spreads across that thin manifold. The obstacle is that $p(\mathbf{x})$ is intractable: we cannot write down a normalized density over $150{,}528$ pixels and integrate it. Every family in this section is a strategy for getting useful behavior out of $p(\mathbf{x})$ without ever writing it down directly. We will tour them in the order they tend to be taught, then assemble a comparison table and a map that the rest of the part hangs on. Each family then gets its own chapter; this section is the index card for all of them.

1. Variational Autoencoders: Compress, Then Decode Beginner

A variational autoencoder (VAE) introduces a low-dimensional latent variable $\mathbf{z}$ with a simple prior (usually a standard Gaussian) and a decoder network $p_\theta(\mathbf{x} \mid \mathbf{z})$ that turns a latent into an image. The marginal likelihood $p_\theta(\mathbf{x}) = \int p_\theta(\mathbf{x} \mid \mathbf{z}) \, p(\mathbf{z}) \, d\mathbf{z}$ is intractable, so the VAE introduces an encoder $q_\phi(\mathbf{z} \mid \mathbf{x})$ and maximizes a tractable lower bound on the log-likelihood, the evidence lower bound or ELBO:

The first term rewards reconstruction; the second, a Kullback-Leibler divergence $D_{\mathrm{KL}}$ (a standard measure of how far one distribution sits from another, zero only when they match), pulls the encoded distribution toward the prior so that sampling $\mathbf{z} \sim p(\mathbf{z})$ and decoding produces plausible images. The VAE gives an approximate, comparable likelihood and a clean, smooth latent space (the subject of Section 30.3), at the cost of samples that are often slightly blurry, because the bound and the Gaussian decoder average over fine detail. Chapter 31 builds the VAE in full, including the reparameterization trick that makes the sampling differentiable.

2. Generative Adversarial Networks: Learn by Being Judged Beginner

A generative adversarial network (GAN) abandons likelihood altogether. A generator $G$ maps a noise vector $\mathbf{z} \sim p(\mathbf{z})$ to an image, and a discriminator $D$ tries to tell real images from generated ones. They play a minimax game:

Read the formula as a tug-of-war over one quantity: the discriminator $D$ turns its dial to make the expression as large as it can (scoring real images near 1 and fakes near 0), while the generator $G$ turns its own dial to make the same expression as small as it can (fooling $D$ into scoring its fakes near 1). Neither term is a loss to minimize on its own; learning is the contest between the two. The generator never sees the data distribution directly; it only learns from the gradient the discriminator provides. When the game reaches equilibrium the generator's samples are indistinguishable from real data. GANs are the family famous for sharp, photorealistic samples and fast single-pass sampling, but they pay for it: they provide no likelihood, training is unstable, and they are prone to mode collapse, where the generator covers only part of the data distribution (high quality, low diversity).

The adversarial discriminator is the spiritual descendant of the realism critic idea you saw used to judge representations in Chapter 25. Chapter 32 develops the game and its many stabilizers.

3. Normalizing Flows: An Exactly Reversible Road Intermediate

A normalizing flow insists on something the other families give up: an exact, computable likelihood. It builds the data distribution as an invertible, differentiable transformation $f$ of a simple base distribution (a Gaussian). Because $f$ is invertible, the change-of-variables formula gives the exact density,

where the Jacobian-determinant term accounts for how the transformation stretches and squeezes volume. That correction term is not optional bookkeeping: probability mass must be conserved, so when $f^{-1}$ compresses a region of pixel space the density there has to rise to keep the total integral at one, and when it expands a region the density must fall. The log-determinant measures exactly that local volume change, and without it the formula would report a base-distribution density that no longer integrates to one over $\mathbf{x}$. This is the same change-of-variables logic that tracked how a homography warps pixel areas back in Chapter 5, now applied to probability mass. Flows let you both sample (push $\mathbf{z}$ through $f$) and evaluate exact likelihood, a rare combination. The price is architectural: every layer must be invertible with an efficiently computable Jacobian determinant. That efficiency requirement is the real constraint, because a general $d \times d$ determinant costs $O(d^3)$, hopeless for $d$ in the millions of pixels. Flows are therefore built from layers whose Jacobian is triangular, so its determinant is just the product of the diagonal entries, an $O(d)$ quantity: coupling layers (split the input, transform one half as a function of the other) and autoregressive layers are the two standard ways to buy invertibility and a cheap determinant at once. This constrains the network design and tends to require many parameters for a given sample quality. Flows are less dominant for raw image generation today, but the invertibility idea reappears in flow matching, the modern continuous-time descendant we flag in Section 30.4 and Chapter 33.

4. Autoregressive Models: One Pixel at a Time Intermediate

An autoregressive model uses the chain rule of probability to factorize the joint distribution over all pixels into a product of conditionals, each pixel depending on the pixels before it in some fixed ordering:

This is the exact analogue of how a language model predicts the next token, applied to pixels (PixelCNN) or, in modern systems, to discrete image tokens from a learned codebook. The factorization is exact, so autoregressive models give an exact likelihood and tend to produce coherent, high-quality samples. Their defining weakness is speed: sampling requires $D$ sequential forward passes, one per pixel or token, so generating a single image can take thousands of steps. The same self-attention machinery from Chapter 22 powers the strongest modern autoregressive image models, which generate sequences of image tokens.

5. Diffusion Models: Reverse the Noise Beginner

A diffusion model defines a fixed forward process that gradually adds Gaussian noise to an image over many steps until it becomes pure noise, then learns a neural network to reverse that process one step at a time. Training reduces to a simple denoising objective: given a noised image and the noise level, predict the noise that was added. Sampling starts from pure noise and applies the learned reverse step repeatedly until an image emerges. Diffusion models combine the high sample quality of GANs with the stable, likelihood-grounded training of the probabilistic families, which is why they dominate image generation in 2024 to 2026. Their historical weakness is sampling speed (many reverse steps), the problem that drove the fast-sampler research wave of Section 30.5. Crucially, diffusion is the destination of the score-and-Langevin machinery we build in Section 30.4: the reverse step is a learned step along the score of a noised distribution. This connection, drawn explicitly here, is what makes Chapter 33 feel like the natural continuation of this chapter rather than a fresh start. The recurring denoising thread, from the Gaussian and non-local-means filters of Chapter 7 to learned denoising autoencoders, culminates in diffusion as iterative learned denoising.

6. The Comparison Table Intermediate

Figure 30.2.1 shows the routes; the table below scores them on the axes that matter in practice. "Tractable likelihood" asks whether you can compute or bound $p(\mathbf{x})$ for a given image. "Sample quality" and "diversity" summarize the family's typical behavior on natural images. "Sampling speed" counts how many network passes one sample costs. The entries are typical tendencies, not laws; a well-engineered member of any family can beat a sloppy member of another.

The code below makes the abstract differences concrete by sketching the one-line sampling signature of each family. The signatures alone reveal the speed story: the families with a single call sample in one pass, while the autoregressive and diffusion families wrap their call in a loop.

# Each family exposes a different sampling SHAPE. The single-pass families return
# after one network call; the autoregressive and diffusion families wrap it in a loop.
# Pseudocode, but the loop structure is exactly what sets per-sample cost.
import torch


def sample_vae(decoder, n, z_dim, device):
    z = torch.randn(n, z_dim, device=device)        # draw from the Gaussian prior
    return decoder(z)                                # ONE pass: latent -> image

def sample_gan(generator, n, z_dim, device):
    z = torch.randn(n, z_dim, device=device)
    return generator(z)                              # ONE pass, just like the VAE decoder

def sample_flow(flow, n, z_dim, device):
    z = torch.randn(n, z_dim, device=device)
    return flow.forward(z)                           # ONE pass through the invertible map

def sample_autoregressive(model, n, D, device):
    x = torch.zeros(n, D, device=device)
    for i in range(D):                               # D SEQUENTIAL passes, one per pixel
        logits = model(x, up_to=i)                   # condition on pixels 0..i-1
        x[:, i] = sample_from(logits)                # then fill pixel i
    return x

def sample_diffusion(denoiser, n, shape, steps, device):
    x = torch.randn(n, *shape, device=device)        # start from pure noise
    for t in reversed(range(steps)):                 # MANY reverse steps
        x = denoiser.reverse_step(x, t)              # each step removes a little noise
    return x

# One call shape for two different families. Whichever pipeline class you load,
# sampling is the same pipe(...) invocation; the per-family reverse loop the
# pseudocode wrote by hand lives inside the pipeline's __call__.
from diffusers import DDPMPipeline, StableDiffusion3Pipeline
# A diffusion model (many reverse steps, handled internally):
pipe = DDPMPipeline.from_pretrained("google/ddpm-cifar10-32").to("cuda")
img = pipe(num_inference_steps=50).images[0]          # the reverse loop is inside .__call__
# A modern rectified-flow text-to-image model, same call shape:
# pipe = StableDiffusion3Pipeline.from_pretrained("stabilityai/stable-diffusion-3.5-medium")
# img = pipe("a watercolor fox").images[0]

7. How to Read the Rest of the Part Beginner

You now have the index card. A one-word verb for each family fixes the order in memory: compress (VAE), contest (GAN), invert (flow), chain (autoregressive), denoise (diffusion); each verb is the route that family takes from a code or from noise to a finished sample. The remaining chapters of Part IV each take one box on this map and open it: Chapter 31 the VAE, Chapter 32 the GAN, Chapter 33 diffusion, with flows and autoregressive models appearing where the story needs them. Two pieces of machinery, the latent space and the score, are shared across boxes and deserve their own treatment before we specialize. The next two sections supply them: Section 30.3 develops the latent variable that the VAE, GAN, and latent-diffusion families all rely on, and Section 30.4 develops the energy and score view that becomes diffusion.