Sampling, Likelihood & the Quality-Diversity-Speed Trilemma

"They asked for pictures that were beautiful, varied, and instant. I drew them two of the three on a napkin and said pick. They kept turning the napkin over hoping a fourth corner would appear. It never does. I have checked many times."

A Generative Model With Opinions About Trade-offs

The map of Section 30.2 told you the families; the latent and score machinery of Section 30.3 and Section 30.4 told you how they work. This section gives you the lens for judging the result. We start with what sampling and likelihood actually mean, show why a model can have excellent likelihood and mediocre-looking samples (or the reverse), then formalize the three-way trade that governs every design decision in the part. By the end you will read each subsequent chapter not as a parade of methods but as a series of moves on one triangle, which is exactly how the field thinks about its own progress.

1. What Sampling Actually Computes Beginner

Sampling means drawing an $\mathbf{x}$ such that, over many draws, the empirical distribution matches the model's $p_\theta(\mathbf{x})$. The cost of one draw, the sampling cost, is the number of expensive operations (network forward passes) it takes, and it varies enormously across families, as the sampling-signature code in Section 30.2 showed. A VAE, GAN, or flow samples in a single pass; an autoregressive model needs one pass per pixel; a diffusion model needs one pass per reverse step. Speed is therefore not an implementation detail but a structural property of the family, set by the shape of the sampling loop. The table below restates the cost in concrete numbers for a typical image, to make the orders of magnitude vivid.

The spread in Table 30.5.1, from one pass to thousands, is three orders of magnitude, and it is the single biggest practical difference between families. It is why a GAN can run as a real-time video filter while a vanilla diffusion model cannot, and why the fast-sampler and distillation rows exist at all: they are the field's response to diffusion's cost. We return to them at the end of the section.

2. Likelihood Is Not Quality Intermediate

A model's likelihood on held-out data, $p_\theta(\mathbf{x})$ for real test images, measures how much probability mass the model puts where the data actually is. It is the natural training objective for the probabilistic families (flows, autoregressive models, the VAE's bound). It is tempting to assume that higher likelihood means better-looking samples. It does not, and the disconnect is one of the most important facts in generative modeling. Theis, van den Oord, and Bethge made the point sharply in 2016: log-likelihood and sample quality are largely independent, and optimizing one is no guarantee of the other.

Two mechanisms drive them apart. First, in high dimensions, average log-likelihood is dominated by getting the bulk statistics and the low-frequency structure right; a model can score well on likelihood while its samples look slightly blurry, because the perceptually crucial high-frequency detail contributes little to the likelihood number. This is the VAE's characteristic failure (good bound, soft samples). Second, a model can produce gorgeous samples while assigning near-zero probability to large regions of the true data, that is, high quality but terrible coverage; a GAN suffering mode collapse is the extreme case, and GANs do not even define a likelihood to expose the problem. The lesson is that you must measure what you care about: if you care about how samples look and how varied they are, likelihood is the wrong yardstick, and the perceptual and distributional metrics of Section 30.6 are the right ones.

3. The Trilemma, Stated Intermediate

Putting the pieces together gives the central organizing principle of the part. Every generator is scored on three desiderata, and improving any two tends to cost you the third:

• Fidelity (quality): do individual samples look like real, sharp, artifact-free images?
• Diversity (coverage): does the model reproduce the full variety of the data distribution, every mode, rather than a favored subset?
• Speed: how few network passes does one sample cost?

This is the quality-diversity-speed trilemma (named as such by Xiao, Kreis, and Vahdat in 2022, in the context of denoising diffusion). Figure 30.5.1 places the families at the trilemma's corners and edges, and the geometry tells the story: each family sits near the two corners it favors and far from the one it sacrifices.

The code below makes the diversity axis concrete by quantifying mode coverage on a toy mixture, the measurement that distinguishes a collapsed generator from a healthy one. It is the simplest possible diversity probe and the conceptual seed of the recall metric in Section 30.6.

# Measure diversity directly: what fraction of the data's modes does a generator
# actually cover? This recall proxy separates a collapsed generator (few modes hit)
# from a healthy one (all modes hit), independent of how sharp each sample looks.
import torch

# A toy data distribution with 8 well-separated modes arranged in a ring.
angles = torch.linspace(0, 2 * torch.pi, 9)[:-1]         # 8 angles around the circle
modes = torch.stack([torch.cos(angles), torch.sin(angles)], dim=1) * 4

def mode_coverage(samples, modes, radius=0.7):
    """Fraction of the 8 modes that have at least one nearby sample (a recall proxy)."""
    d = torch.cdist(samples, modes)                  # (n_samples, 8) distances
    covered = (d.min(dim=0).values < radius)         # is each mode hit by some sample?
    return covered.float().mean().item()

# A healthy generator hits all modes; a collapsed one hits only a few.
healthy   = modes[torch.randint(0, 8, (500,))] + 0.1 * torch.randn(500, 2)
collapsed = modes[torch.randint(0, 2, (500,))] + 0.1 * torch.randn(500, 2)  # only 2 modes
print("healthy coverage:  ", mode_coverage(healthy, modes))     # healthy coverage:   1.0
print("collapsed coverage:", mode_coverage(collapsed, modes))   # collapsed coverage: 0.25

4. Why Diffusion's Speed Problem Launched a Field Intermediate

Diffusion models landed in the enviable position of Figure 30.5.1: top fidelity and full coverage, the two corners hardest to get together. Their single weakness was the speed corner, vanilla sampling needed hundreds to a thousand network passes per image. Because the other two corners were already won, an enormous research effort poured into the third, and it is worth knowing the shape of that effort because it organizes much of Chapter 33. The attacks come in two flavors. Better solvers treat sampling as numerically integrating a differential equation (the score-SDE or its deterministic ODE form) and use higher-order integrators (DDIM, DPM-Solver) to take far fewer, larger steps, cutting a thousand passes to twenty or fifty with little quality loss. Distillation trains a student network to reproduce in one or a few passes what the teacher diffusion model does in many, the route of progressive distillation and consistency models, pushing toward the single-pass speed of a GAN while keeping diffusion's coverage.

# Move along the speed axis of the trilemma without retraining: swap the scheduler.
# The same trained model sampled with a higher-order solver at 20 steps runs roughly
# fifty times faster than a basic sampler at ~1000 steps, at similar quality.
from diffusers import DDPMScheduler, DPMSolverMultistepScheduler

# Slow but simple: hundreds to a thousand steps.
slow = DDPMScheduler.from_pretrained("model-id")          # ~1000 passes per image
# Fast solver: comparable quality in far fewer steps.
fast = DPMSolverMultistepScheduler.from_pretrained("model-id")
# pipe.scheduler = fast; pipe(num_inference_steps=20)     # ~20 passes per image

5. The Compass for the Rest of the Part Beginner

You now hold the practitioner's compass. Sampling cost is structural, set by the family; likelihood and visual quality are different things that can move in opposite directions; and fidelity, diversity, and speed form a triangle on which every generator and every advance can be placed. When you read Chapter 32 on GANs, watch the diversity corner (mode collapse and its fixes); when you read Chapter 33 on diffusion, watch the speed corner. The only remaining foundational tool is how to actually measure the fidelity and diversity axes for real images, since the eye-test is not enough, as the medical-imaging story showed. That measurement problem is the subject of the final section.