The Score-Based View: VE/VP SDEs & the Probability-Flow ODE

"Tell me which way the density rises and I will climb it forever. That gradient is the only compass I have ever needed, and the only one I ever trusted."

A Score Function Pointing Uphill

The DDPM of Section 33.2 is a discrete chain of $T$ steps. This section asks what happens as $T \to \infty$ and the steps become infinitesimal. If the stochastic calculus on the next few lines looks like a detour into unrelated math, here is the reason to push through: it ends with a small shock. The noise-prediction network you already trained in Section 33.1 turns out to be, up to a single rescaling, the score estimator from Chapter 30; you built two famous models at once without knowing it, and this section is where the disguise comes off.

The plan is to take that one shock apart into three results, each developed in its own subsection. First, the continuous limit turns the forward chain into a stochastic differential equation, and its time reversal contains exactly one unknown, the score function of Chapter 30, which is what reveals that your noise predictor was a score estimator all along. Second, there are two natural ways to set up the forward SDE (variance-exploding and variance-preserving), and naming them lets you read any model's configuration. Third, every diffusion model has a hidden deterministic counterpart, the probability-flow ODE, that the rest of the chapter relies on heavily for fast and reproducible sampling. Take the three in order and the stochastic-calculus notation stays grounded in objects you already built.

1. From Discrete Steps to a Continuous SDE Advanced

A stochastic differential equation describes how a quantity evolves under a deterministic push plus continuous random noise. Written in the standard form, the forward diffusion is

where $f(x, t)$ is the drift (a deterministic vector field pulling the sample somewhere), $g(t)$ is the diffusion coefficient (how much noise is injected per unit time), and $dW$ is the increment of a Wiener process, the continuous-time analogue of adding a fresh Gaussian at each step. The DDPM update from Section 33.2, $x_t = \sqrt{1-\beta_t}\,x_{t-1} + \sqrt{\beta_t}\,z$, is exactly the Euler discretization of such an SDE; in the limit of small steps it becomes a specific $f$ and $g$. The forward SDE is not learned. The whole point is what happens when you reverse it.

2. The Reverse SDE and the Score Advanced

A 1982 result by Anderson states that any forward SDE has a time-reversed SDE, and gives its form explicitly. The reverse of the equation above is

where $p_t(x)$ is the distribution of noisy samples at time $t$ and $d\bar W$ is a reverse-time Wiener increment. Everything in this equation is known except one term: $\nabla_x \log p_t(x)$, the score, the gradient with respect to the image of the log-density of noisy data. This is the same score you met in the energy-based models and score matching of Chapter 30; there it was the gradient that Langevin dynamics climbed, here it is the only unknown standing between us and a generator. If we can estimate the score at every noise level, we can simulate the reverse SDE and turn noise into data. Figure 33.3.1 shows how the forward SDE flattens a structured density into a Gaussian and the reverse SDE, steered by the score, rebuilds it. The illustration below makes the score concrete as a compass that always points uphill toward the data.

2b. VE and VP: Two Ways to Add Noise Advanced

The forward SDE has two canonical instantiations, distinguished by what they do to the total variance of $x_t$.

• Variance Exploding (VE): the drift is zero and the noise scale $g(t)$ grows without bound, so $x_t = x_0 + \sigma(t)\,\epsilon$ with $\sigma(t)$ rising to a very large value. The variance "explodes" because nothing shrinks the signal; corruption is achieved by drowning it in ever-larger noise. This is the formulation of the original score-matching models, the noise conditional score network (NCSN).
• Variance Preserving (VP): the drift shrinks the signal as noise is added, exactly the $\sqrt{1-\beta_t}$ factor of DDPM, so the total variance stays near one throughout. DDPM is the discrete VP SDE.

The two are related by a simple change of variables and produce equivalent models, but they differ in numerical conditioning and in how the noise levels are chosen, which is why a sampler written for one needs care to apply to the other. The EDM framework of Section 33.2 reparameterizes both in terms of a single noise scale $\sigma$, which is the cleanest way to hold them in one head. The practical upshot: when you read that a model uses a "VP schedule" it is a DDPM-style process; "VE" means the NCSN-style explode-the-variance process; and EDM's $\sigma$-parameterization subsumes both.

3. The Probability-Flow ODE Advanced

Here is the result that the rest of the chapter leans on. For every forward SDE there exists an ordinary differential equation, with no noise term at all, whose solution trajectories have the same marginal distributions $p_t(x)$ at every time as the noisy SDE. It is called the probability-flow ODE:

Compare this to the reverse SDE in subsection 2: the score appears again, but the noise injection $g(t)\,d\bar W$ is gone and the score term carries a factor of one half instead of one. Because there is no randomness, this ODE defines a deterministic, invertible map between a noise sample and a data sample. Two consequences make it central. First, deterministic trajectories are smooth, so you can solve them with large, accurate steps using standard ODE solvers, which is the entire basis of fast sampling in Section 33.4. Second, the map being invertible means you can run it forward to encode a real image into its corresponding noise, the foundation of the inversion-based editing you will meet in Chapter 35. The code below solves the probability-flow ODE for the trained 2D denoiser using a simple Euler integrator over the $\sigma$-schedule.

# Deterministic sampling via the probability-flow ODE.
# No noise is drawn inside the loop: at each step we predict x0 from the
# rescaled score, then move to the next lower noise level on a coarse grid.
import torch

@torch.no_grad()
def pf_ode_sample(model, alpha_bars, n=512, steps=50, T=200):
    """Deterministic sampling by Euler-integrating the probability-flow ODE."""
    x = torch.randn(n, 2)                                   # start from noise
    ts = torch.linspace(T - 1, 0, steps).long()             # coarse time grid
    for i in range(len(ts) - 1):
        t, t_next = ts[i], ts[i + 1]
        ab, ab_next = alpha_bars[t], alpha_bars[t_next]
        eps = model(x, t.repeat(n))                         # eps_theta = score (rescaled)
        # predict x0, then move deterministically toward the next (lower) noise level
        x0_hat = (x - (1 - ab).sqrt() * eps) / ab.sqrt()
        x = ab_next.sqrt() * x0_hat + (1 - ab_next).sqrt() * eps
    return x

# samples = pf_ode_sample(model, alpha_bars)   # 50 deterministic steps, no torch.randn inside
print("probability-flow ODE: deterministic, ~50 steps instead of 200")

4. The Forward SDE in Full: VE and VP Kernels Advanced

Subsection 2b named the two families; a graduate course needs their exact drift, diffusion, and transition kernels, because those closed forms are what you actually simulate and what the loss is computed against. Start again from the general forward SDE, now written with the Wiener increment as $dw$ to match the primary source (Song et al., 2021):

A linear SDE of this form has a Gaussian transition kernel $p_{0t}(x_t \mid x_0) = \mathcal N(x_t; \mu_t(x_0), \Sigma_t)$, whose mean and covariance solve a pair of ordinary differential equations (the moment equations of the SDE). We solve those for each family.

4.1 Variance Exploding (VE)

The VE SDE is the continuous limit of the score-matching models (SMLD/NCSN), in which corruption is pure noise injection with no shrinkage of the signal. It sets the drift to zero and chooses the diffusion coefficient so that the running variance tracks a prescribed schedule $\sigma^2(t)$:

With zero drift the mean does not move, so $\mu_t(x_0) = x_0$. The variance accumulates the squared diffusion coefficient: $\frac{d}{dt}\Sigma_t = g(t)^2 I = \frac{d}{dt}\sigma^2(t)\, I$, which integrates to $\Sigma_t = [\sigma^2(t) - \sigma^2(0)]\, I$. The transition kernel is therefore

The name is now literal: as $t$ grows, $\sigma(t)$ is driven to a very large value, so the variance explodes while the mean stays pinned at $x_0$. The signal is never attenuated; it is simply drowned. Sampling from this kernel is one line, $x_t = x_0 + \sqrt{\sigma^2(t) - \sigma^2(0)}\,\epsilon$ with $\epsilon \sim \mathcal N(0, I)$, which is exactly the multi-scale noise corruption of NCSN.

4.2 Variance Preserving (VP) and Recovering DDPM

The VP SDE is the continuous limit of DDPM. Here the drift actively shrinks the signal as noise is added so the total variance is held near one, which is what makes the inputs to the network stay on a fixed scale across all noise levels. With a noise-rate schedule $\beta(t) > 0$,

This is a linear SDE with $f(x,t) = -\tfrac12 \beta(t) x$ and $g(t) = \sqrt{\beta(t)}$. The mean obeys $\frac{d}{dt}\mu_t = -\tfrac12 \beta(t)\mu_t$, a scalar linear ODE whose solution is an exponential decay, and the variance obeys $\frac{d}{dt}\Sigma_t = -\beta(t)\Sigma_t + \beta(t) I$, a linear ODE driven toward the identity. Solving both with $\mu_0 = x_0$ and $\Sigma_0 = 0$ gives the closed-form marginal

The mean decays toward zero and the variance rises toward one, so their squares always sum to one (in the per-coordinate, unit-data-variance sense): the variance is preserved. Deriving these two formulas from the SDE is Exercise 33.3.4. The intuition behind the $-\tfrac12 \beta x$ drift is precisely this balance: the exponent on the mean is half the exponent on the variance, which is what forces $\mu_t^2 + \Sigma_t = 1$.

5. Deriving the Reverse-Time SDE and Its Training Objective Advanced

Subsection 2 stated Anderson's reverse-time SDE; here is what it says and how the single unknown becomes a tractable regression target. Anderson (1982) proved that if a process evolves forward by $dx = f(x,t)\,dt + g(t)\,dw$, then running time backward the same marginals $p_t$ are reproduced by another diffusion:

where $d\bar w$ is a Wiener increment for time flowing in reverse (an infinitesimal Gaussian as $t$ decreases from $T$ to $0$), and $\nabla_x \log p_t(x)$ is the score of the noisy marginal. Read the drift term by term: $f(x,t)$ is the original forward push, which in reverse time still acts, and $-g(t)^2 \nabla_x \log p_t(x)$ is the correction that bends trajectories back uphill toward regions of high data density. The diffusion coefficient $g(t)$ is unchanged from the forward process; only the drift acquires the score.

Every quantity here is known except the score $\nabla_x \log p_t(x)$, because $p_t$ is the marginal of the data pushed through the forward SDE and we have no closed form for it. We learn a network $s_\theta(x, t)$ to approximate it. The marginal score is intractable, but the per-sample conditional score $\nabla_{x_t} \log p_{0t}(x_t \mid x_0)$ is available in closed form (it is the Gaussian kernel of subsection 4), and a classical identity (Vincent, 2011) shows that regressing onto the conditional score also fits the marginal score in expectation. The training objective is weighted denoising score matching:

The outer expectation samples a time $t$ (uniformly or from a chosen distribution), $\lambda(t) > 0$ is a positive weighting that balances the loss across noise levels, $x_0$ is drawn from data, and $x_t$ is drawn from the forward kernel given $x_0$. For the VP kernel the conditional score is $\nabla_{x_t}\log p_{0t}(x_t\mid x_0) = -(x_t - \mu_t)/\Sigma_t = -\,\epsilon/\sqrt{1-\bar\alpha_t}$, which is precisely the noise-prediction target of Section 33.1 up to the $1/\sqrt{1-\bar\alpha_t}$ rescaling of the Key Insight in subsection 2. Choosing $\lambda(t) = \Sigma_t$ (the kernel variance) turns this objective into the plain noise-prediction MSE that DDPM minimizes. The same loss, two weightings, two communities.

6. Deriving the Probability-Flow ODE from Fokker-Planck Advanced

Subsection 3 wrote the probability-flow ODE down and described its consequences; the course needs the derivation, which is a short and elegant manipulation of the Fokker-Planck equation. The point to establish is that a deterministic flow can reproduce the exact same time-varying density $p_t$ as the noisy SDE, with no randomness at all.

Any SDE $dx = f(x,t)\,dt + g(t)\,dw$ induces a partial differential equation for how its marginal density $p_t(x)$ evolves, the Fokker-Planck (forward Kolmogorov) equation. With a scalar diffusion coefficient $g(t)$ it reads

The first term is transport by the drift and the second is diffusion (a Laplacian smoothing). The trick is to rewrite the diffusion term as the divergence of a flux so the whole right-hand side becomes a single divergence, which is the signature of a deterministic transport (continuity) equation. Use the identity $\nabla^2 p_t = \nabla\cdot(\nabla p_t)$ and the log-derivative identity $\nabla p_t = p_t\, \nabla \log p_t$:

Substituting back and pulling both terms under one divergence,

This is exactly the continuity (Liouville) equation $\partial_t p_t + \nabla\cdot(\tilde f\, p_t) = 0$ for particles carried by the deterministic velocity field

A continuity equation says: if you move every particle along the ODE $\frac{dx}{dt} = \tilde f(x,t)$, the population density they carry evolves by precisely this PDE. Since that PDE is identical to the Fokker-Planck equation of the original SDE, the deterministic flow and the stochastic process share the same marginals $p_t(x)$ at every time. That deterministic flow is the probability-flow ODE:

6.1 Exact Likelihoods via the Instantaneous Change of Variables

Because the probability-flow ODE is a deterministic, invertible map, it is a continuous normalizing flow, and continuous normalizing flows come with an exact likelihood. Along an ODE trajectory the log-density evolves by the instantaneous change-of-variables formula:

Integrating this from $t = 0$ to $t = T$ along the trajectory that starts at a data point $x_0$ gives $\log p_0(x_0)$ as the known Gaussian log-density at $x_T$ minus the accumulated divergence. The one obstacle is that the divergence $\nabla\cdot\tilde f$ is a trace of a $d\times d$ Jacobian, expensive for image-sized $d$. The Hutchinson trace estimator replaces it with a stochastic estimate $\nabla\cdot\tilde f = \mathbb E_{v}[\, v^\top \partial_x \tilde f\, v\,]$ for $v$ with identity covariance (for example Rademacher or standard Gaussian), and $v^\top \partial_x \tilde f$ is a single vector-Jacobian product cheaply obtained by automatic differentiation. This is how the SDE paper reports exact bits-per-dimension likelihoods for diffusion models, a number GANs cannot produce at all.

7. Predictor-Corrector Sampling Advanced

The reverse SDE and the probability-flow ODE both give samplers, but each leaves something on the table: a numerical SDE solver accumulates discretization error in the drift, and once the iterate has drifted off the true $p_t$ there is nothing to pull it back. Predictor-corrector (PC) sampling fixes this by interleaving two moves. The predictor takes one step of a reverse-time numerical solver (for instance an Euler-Maruyama step of the reverse SDE) to advance the noise level. The corrector then holds the time $t$ fixed and runs several steps of score-based MCMC to nudge the sample back onto $p_t$, using only the learned score. The natural corrector is annealed Langevin dynamics, the same sampler that powered NCSN in Chapter 30:

Each Langevin step takes a small ascent of size $\epsilon$ along the score (uphill toward higher density) and adds calibrated noise $\sqrt{2\epsilon}\, z$; the $\sqrt 2$ is exactly the amount that makes $p_t$ the stationary distribution of the chain, so iterating leaves the sample correctly distributed. Predictor and corrector use the very same network $s_\theta$, so the corrector is free in engineering terms; it just trades a few extra function evaluations for samples that sit more faithfully on the marginal. The result is the predictor-corrector sampler that gave the SDE framework its strongest sample quality.

8. A From-Scratch Implementation Advanced

The derivations above become concrete in a few lines of PyTorch. The snippet implements the VP-SDE forward marginal sampler of subsection 4.2 (drawing $x_t$ from its closed-form Gaussian given $x_0$) and a single Euler step of the probability-flow ODE of subsection 6 (with the correct $\tfrac12 g^2$ factor on the score), given any score network $s_\theta$.

import torch

def vp_alpha_bar(t, beta_min=0.1, beta_max=20.0):
    # integral of beta(s) for the standard linear VP schedule, t in [0, 1]
    integral = beta_min * t + 0.5 * (beta_max - beta_min) * t**2
    return torch.exp(-integral)              # = exp(-int_0^t beta(s) ds)

def vp_forward_sample(x0, t):
    # closed-form VP marginal: mean x0 * sqrt(abar), var (1 - abar)
    abar = vp_alpha_bar(t).view(-1, *([1] * (x0.dim() - 1)))
    eps = torch.randn_like(x0)
    xt = abar.sqrt() * x0 + (1 - abar).sqrt() * eps
    return xt, eps

def beta(t, beta_min=0.1, beta_max=20.0):
    return beta_min + (beta_max - beta_min) * t

def pf_ode_euler_step(score_net, x, t, dt):
    # probability-flow ODE: dx/dt = f - 0.5 g^2 score,  f = -0.5 beta x,  g^2 = beta
    b = beta(t).view(-1, *([1] * (x.dim() - 1)))
    drift = -0.5 * b * x - 0.5 * b * score_net(x, t)   # note the 1/2 on the score
    return x + drift * dt                               # Euler step (dt < 0 when sampling)