Guidance: Classifier & Classifier-Free

"Left to my own devices I will draw you something plausible. Whisper a label in my ear and turn the dial up, and I will draw you something unmistakable, occasionally at the cost of subtlety."

A Conditional Generator With a Volume Knob

So far the chapter has built unconditional generators that sample some image from the data distribution. Useful generation is almost always conditional: you want a photo of a specific class, or an image matching a text prompt. Section 33.2 showed how to condition the network by feeding it an extra input, the network simply takes the class or text embedding alongside the noisy image. But conditioning alone is often too weak; the model treats the condition as a gentle suggestion and produces only loosely related images. Guidance is the technique that amplifies the condition's influence. We start with the original classifier guidance to build intuition from the score, then derive classifier-free guidance, the method that powers modern systems, and discuss the fidelity-diversity trade-off the guidance scale controls.

1. Conditioning vs Guidance Intermediate

To condition a diffusion model, you give the denoiser the condition $c$ as an additional input: $\epsilon_\theta(x_t, t, c)$. For a class label, $c$ is an embedding added to the timestep embedding; for text, $c$ is a sequence of token embeddings consumed by the cross-attention layers of the U-Net, the same attention mechanism from Chapter 22, where the image features form the queries and the text tokens form the keys and values. This is genuine conditioning, and a well-trained conditional model does respond to $c$. The problem is calibration: maximum-likelihood training spreads probability over everything consistent with $c$, which is a lot, so samples often capture the gist of the condition but not its full specificity. Guidance sharpens the conditional distribution, biasing samples toward regions where the condition is strongly satisfied, at some cost to diversity. The next two subsections give two ways to do it.

2. Classifier Guidance Advanced

The score view of Section 33.3 makes guidance almost obvious. We want to sample from the conditional distribution $p(x \mid c)$, whose score is $\nabla_x \log p(x_t \mid c)$. By Bayes' rule, $\log p(x_t \mid c) = \log p(x_t) + \log p(c \mid x_t) - \log p(c)$, and taking the gradient (the last term vanishes since it does not depend on $x$):

The first term is the score the diffusion model already estimates (via its noise prediction, per the identity in Section 33.3), the same gradient-of-log-density object you met in the energy-based models of Chapter 30. The second term is the gradient of a classifier that predicts the condition from the noisy image. So Dhariwal and Nichol's classifier guidance trains a separate classifier on noisy images and, at each sampling step, adds a multiple $s$ of its gradient to the score:

The formula is the score equation above, just rewritten in noise terms: the minus sign and the $\sqrt{1-\bar\alpha_t}$ factor are exactly the score-to-noise conversion $s_\theta = -\epsilon_\theta/\sqrt{1-\bar\alpha_t}$ from Section 33.3 applied to both terms, so "add the classifier gradient to the score" becomes "subtract a scaled gradient from the predicted noise." The guidance scale $s$ controls how hard the classifier pushes. This works and was how the first diffusion models beat GANs on conditional ImageNet, but it has two real drawbacks: you must train and maintain a separate classifier, and that classifier has to be trained on noisy images at every noise level, which off-the-shelf classifiers are not. That friction motivated the alternative that replaced it.

3. Classifier-Free Guidance Intermediate

Classifier-free guidance (CFG) removes the external classifier entirely with a beautiful trick. During training, the model is shown the condition most of the time but, with some probability (say 10 to 20 percent), the condition is dropped and replaced by a special null token $\varnothing$. The single network thus learns both the conditional noise $\epsilon_\theta(x_t, t, c)$ and the unconditional noise $\epsilon_\theta(x_t, t, \varnothing)$. At sampling time, you compute both and extrapolate: push away from the unconditional prediction in the direction of the conditional one.

Here $w$ is the guidance scale (in the diffusers and Stable Diffusion world it is the guidance_scale parameter, typically 5 to 12 for text-to-image). At $w = 1$ you recover plain conditional sampling; $w = 0$ is unconditional; larger $w$ exaggerates the difference between conditional and unconditional, sharpening adherence to the prompt. The reason this works is that the difference $\epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, \varnothing)$ is an implicit estimate of the classifier gradient $\nabla_x \log p(c \mid x_t)$ from subsection 2, obtained without any classifier. Figure 33.6.1 shows the extrapolation geometrically.

The function below implements that extrapolation: it runs the conditional and unconditional predictions in one batched pass and combines them with the guidance scale, the drop-in replacement for the plain noise estimate in any sampler.

# Classifier-free guidance at sampling time.
# Run the unconditional and conditional noise predictions in one batched
# pass, then extrapolate beyond the conditional by guidance_scale.
import torch

@torch.no_grad()
def cfg_predict(unet, x_t, t, cond_emb, null_emb, guidance_scale=7.5):
    """Classifier-free guidance: one network, two conditionings, extrapolate."""
    # Batch the conditional and unconditional inputs together for one forward pass
    x_in = torch.cat([x_t, x_t], dim=0)
    t_in = torch.cat([t, t], dim=0)
    c_in = torch.cat([null_emb, cond_emb], dim=0)        # uncond first, cond second
    eps_uncond, eps_cond = unet(x_in, t_in, c_in).sample.chunk(2)
    # Extrapolate away from the unconditional toward the conditional
    return eps_uncond + guidance_scale * (eps_cond - eps_uncond)

# To DISABLE the condition during training, randomly replace cond_emb with null_emb:
# if torch.rand(1) < 0.1: cond_emb = null_emb     # ~10% null-conditioning dropout

4. Classifier Guidance, Derived Rigorously Advanced

Subsection 2 stated the classifier-guidance formula and gave its intuition. Here we derive it carefully, because the derivation exposes exactly what the guidance scale $s$ does to the sampled distribution and why $s = 1$ is special. The setup is a deliberately reshaped target. Instead of sampling the true posterior $p(x \mid y)$ for a condition $y$, classifier guidance samples from a tempered posterior

where the classifier likelihood $p(y \mid x)$ is raised to a power $s \ge 0$ before being combined with the unconditional prior $p(x)$. The exponent $s$ is the only free parameter. When $s = 1$ this is exactly Bayes' rule, $p(x)\,p(y \mid x) \propto p(x \mid y)$, so guidance with $s = 1$ samples the true Bayesian posterior and nothing more. When $s > 1$ the likelihood term is sharpened: regions where the classifier is confident about $y$ get their probability raised to a power greater than one and so dominate even more, which concentrates samples onto the most unambiguous exemplars of $y$ at the cost of diversity. When $s < 1$ the condition is softened toward the unconditional prior. The single scalar is therefore a temperature on the classifier, not a probability, which is the reason turning it up does not give "more correct" samples but rather "more stereotypical, less varied" ones.

Take the log and the gradient with respect to $x$ of the tempered posterior. The normalizing constant does not depend on $x$ and drops, leaving the guided score in its cleanest form:

Each term is now unambiguous. The first, $\nabla_x \log p_t(x)$, is the unconditional score the diffusion model already supplies. The second, $\nabla_x \log p_t(y \mid x)$, is the gradient of the log-likelihood of a classifier $p_\phi$ evaluated at the noisy sample $x_t$, and $s$ scales how hard that gradient pushes. Setting $s = 1$ recovers the plain Bayes posterior of subsection 2; the extra factor $s$ is precisely the exponent on the likelihood. This is the score-domain statement. To use it inside an $\epsilon$-prediction sampler we convert via the identity from Section 33.3, $\nabla_x \log p_t(x) = -\,\epsilon_\theta(x_t, t)/\sqrt{1 - \bar\alpha_t}$, which says the predicted noise is the negative score scaled by $\sqrt{1 - \bar\alpha_t}$. Multiplying the whole guided score by $-\sqrt{1 - \bar\alpha_t}$ to express it as an effective noise prediction gives the form you implement:

Read term by term: start from the network's noise prediction $\epsilon_\theta(x_t, t)$, then subtract the classifier gradient $\nabla_{x_t} \log p_\phi(y \mid x_t)$, weighted by the guidance scale $s$ and by $\sqrt{1 - \bar\alpha_t}$. The minus sign is the score-to-noise sign flip; the $\sqrt{1 - \bar\alpha_t}$ factor is the same conversion factor applied to the classifier term so that both pieces live in noise units and can be added. Plug $\hat\epsilon$ into any DDPM or DDIM update from Section 33.4 in place of $\epsilon_\theta$ and the sampler now walks toward the tempered posterior.

5. Classifier-Free Guidance: Two Conventions and the Implicit Classifier Advanced

Subsection 3 introduced classifier-free guidance (CFG) and gave one form of its update. There are two widely used algebraic forms of the same operation, and they assign different meanings to the guidance scalar. Confusing them is the single most common source of "my guidance scale of 7.5 behaves like someone else's 6.5" bugs, so we state both explicitly and pin down the conversion. Ho and Salimans train one network jointly on the conditional objective and, with some dropout probability, on the unconditional objective by replacing the condition with a null token $\varnothing$. The same network thus estimates both $\epsilon_\theta(x_t, c)$ and $\epsilon_\theta(x_t, \varnothing)$. Their sampling rule, in the paper's own notation, is

Here $w = 0$ means no guidance (you get the plain conditional prediction), and $w > 0$ extrapolates. Most implementations, including the diffusers library and Stable Diffusion, instead expose a "guidance scale" $s$ and write the algebraically rearranged form

In this form $s = 1$ means no guidance (the unconditional plus one times the difference collapses to the plain conditional), and $s > 1$ extrapolates. The two are the same equation: expanding the second gives $(1)\epsilon_\varnothing + s\,\epsilon_c - s\,\epsilon_\varnothing = s\,\epsilon_c - (s-1)\epsilon_\varnothing$, which matches the first under the relation $s = 1 + w$. The familiar default $s = 7.5$ corresponds to $w = 6.5$.

Why does extrapolating between two noise predictions do anything coherent? Convert both to scores and read the result as a probability statement. Using $\epsilon \propto -\nabla \log p$, the implementation form becomes, in score terms,

The bracketed difference is the gradient of $\log p(x \mid c) - \log p(x) = \log\frac{p(x \mid c)}{p(x)} = \log p(c \mid x) + \text{const}$, by Bayes' rule. So the difference between the conditional and unconditional scores is the gradient of an implicit classifier $p(c \mid x) \propto p(x \mid c)/p(x)$, obtained without ever training one. CFG is therefore classifier guidance with the classifier replaced by the difference of the two predictions the single network already makes, and the scale $w$ sharpens that implicit classifier exactly as $s$ sharpened the explicit one in subsection 4. This is the precise sense in which CFG "needs no separate classifier."

6. Diffusion for Inverse Problems Advanced

Guidance steers generation toward a label or a prompt. A closely related and practically enormous problem is steering generation to be consistent with a measurement: given a blurred, masked, downsampled, or otherwise degraded observation, reconstruct a plausible clean image. This is an inverse problem, and a pretrained unconditional diffusion model is a powerful image prior for it, no retraining required. Model the measurement as

where $x_0$ is the unknown clean image, $\mathcal A$ is a known forward operator (a blur kernel, a masking operator for inpainting, a downsampling operator for super-resolution), and $n$ is measurement noise of variance $\sigma_y^2$. We want to sample the posterior $p(x_0 \mid y)$. As in classifier guidance, the posterior score splits by Bayes' rule into the prior score the diffusion model gives us plus a measurement-likelihood term:

The first term is supplied by the unconditional model. The hard term is the second, the likelihood of the measurement given the noisy intermediate $x_t$, because $y$ is a function of the clean $x_0$, not of $x_t$. There is no closed form for $p_t(y \mid x_t)$, since it requires marginalizing over all clean images consistent with $x_t$. Diffusion Posterior Sampling (DPS) makes this tractable with one well-chosen approximation.

The key tool is Tweedie's formula, which gives the posterior mean of the clean image given the noisy one directly from the score. For the variance-preserving forward process, the minimum-mean-square estimate of $x_0$ given $x_t$ is

where $s_\theta(x_t, t) \approx \nabla_{x_t} \log p_t(x_t)$ is the learned score. Intuitively, $\hat x_0$ is the model's single best guess at the clean image hidden inside $x_t$: it takes the noisy sample, nudges it along the score toward higher density, and rescales by $1/\sqrt{\bar\alpha_t}$ to undo the forward shrinkage. DPS now approximates the intractable $p_t(y \mid x_t)$ by collapsing the distribution over clean images to this point estimate, $p_t(y \mid x_t) \approx p(y \mid \hat x_0(x_t))$. Under the Gaussian measurement model that likelihood is $p(y \mid \hat x_0) \propto \exp\!\big(-\frac{1}{2\sigma_y^2}\|y - \mathcal A(\hat x_0)\|^2\big)$, so its log-gradient is

Note that $\hat x_0$ itself depends on $x_t$ through the network, so this gradient is taken through the score network by automatic differentiation, a vector-Jacobian product, not a gradient of a fixed quantity. In practice DPS folds the $1/(2\sigma_y^2)$ and the inevitable approximation error into a single tunable step size $\zeta_i$ and applies the correction as an extra term after each ordinary reverse step. If $x_{i-1}'$ is the sample produced by one unconditional reverse-diffusion update, the DPS update is

This is remarkably general: any forward operator $\mathcal A$ you can differentiate, linear or nonlinear, plugs in directly, which is why DPS handles inpainting, deblurring, super-resolution, and phase retrieval with the same code and a single pretrained prior.

DPS's point-estimate approximation, replacing the full clean-image posterior by its mean $\hat x_0$, is its main weakness, and it ignores how measurement noise interacts with the operator. The follow-up $\Pi$GDM (pseudoinverse-guided diffusion models) sharpens the likelihood for linear operators $H$ by accounting for both the diffusion uncertainty at time $t$ (a variance $r_t^2$) and the measurement noise $\sigma_y^2$, replacing the isotropic $1/\sigma_y^2$ weighting with the noise-aware matrix $(r_t^2 H H^\top + \sigma_y^2 I)^{-1}$. This weights each measurement direction by how much of its uncertainty comes from the prior versus the sensor, which improves reconstruction quality on linear problems at the cost of needing the operator's structure $H$ explicitly rather than just a black-box $\mathcal A$.