Real-Image Inversion & Faithful Editing

"To edit your photograph, I must first imagine the exact noise I would have hallucinated to paint it. It is like reverse-engineering a dream from the bedsheets."

A Diffusion Model Practicing Forensic Reconstruction

Every editing method so far quietly assumed the image was either generated by the model (so its noise is known) or could be roughly approximated. Real photographs are neither. A photo was made by a camera, not by the model, and to bring it under the model's control you must answer a hard question: what noise would this model have started from to produce this exact image? This section is about answering that question well enough to edit a real photo and have the edit change only what you intended. It is the most technically demanding section of the chapter and the one that makes everything before it work on real images. The illustration below frames inversion as exactly this forensic reconstruction.

1. Why Encoding Is Not Inversion Intermediate

A natural first thought: to edit a real image, just encode it with the VAE of Chapter 31 to get its latent, add some noise, and denoise with a new prompt (this is the strength-based image-to-image of many tools). This works for heavy stylization but fails for precise editing, because adding random noise and denoising is a lossy, stochastic round trip: it does not return the original image, it returns a nearby image the model finds plausible. Faces shift, text scrambles, fine detail is reinvented. To edit faithfully you need the specific noise that regenerates this image, not just any noise of the right magnitude. The key insight below makes the stakes of that distinction concrete.

Recovering that specific noise is inversion. The key enabling fact is that the DDIM sampler of Section 33.4 is deterministic: given a latent and the predicted noise, the reverse step is a fixed function with no random draw. A deterministic map can, in principle, be run in either direction. The DDIM update that denoises from $z_t$ to $z_{t-1}$ can be rearranged to noise from $z_{t-1}$ up to $z_t$, walking the trajectory backward from the clean image toward noise. Figure 35.5.1 contrasts the lossy and the inversion routes.

2. DDIM Inversion Advanced

The DDIM forward (denoising) step predicts the clean image $\hat{z}_0$ from $z_t$ using the network's noise estimate, then takes a deterministic step to $z_{t-1}$. Inversion reverses this: it assumes the noise prediction at $z_{t-1}$ is approximately the same as at $z_t$ (true in the limit of small steps) and solves for $z_t$ given $z_{t-1}$, marching from $t=0$ (the clean encoded image) up to $t=T$ (the recovered noise). The update, with $\bar\alpha_t$ the cumulative product from Chapter 33, is the DDIM step read in the increasing-$t$ direction. The function below implements exactly that march.

# DDIM inversion: recover the noise latent z_T that, denoised deterministically,
# regenerates a given clean latent z0. It runs the DDIM step in the increasing-t
# direction, reusing the same noise prediction the local-linearity assumption allows.
import torch

@torch.no_grad()
def ddim_invert(z0, model, scheduler, prompt_embeds, num_steps=50):
    """Walk a clean latent z0 backward to recover the noise that regenerates it."""
    scheduler.set_timesteps(num_steps, device=z0.device)
    timesteps = reversed(scheduler.timesteps)          # go from low t to high t
    z = z0
    for i, t in enumerate(timesteps):
        eps = model(z, t, encoder_hidden_states=prompt_embeds).sample   # noise pred
        ab_t = scheduler.alphas_cumprod[t]
        t_next = timesteps[i + 1] if i + 1 < len(timesteps) else t
        ab_next = scheduler.alphas_cumprod[t_next]
        # Predict x0, then re-noise to the NEXT (higher) timestep: the inverse step.
        z0_pred = (z - (1 - ab_t).sqrt() * eps) / ab_t.sqrt()
        z = ab_next.sqrt() * z0_pred + (1 - ab_next).sqrt() * eps
    return z                                            # approx. latent noise z_T

This inversion is accurate when generation uses no guidance. But practical editing wants a guidance scale well above 1, and there the inversion breaks: the noise prediction used during inversion (computed without strong guidance) no longer matches the guided prediction used during generation, so the trajectories diverge and the reconstruction drifts. This guidance gap is the central obstacle to faithful real-image editing, and it is what the next method exists to close.

3. Null-Text Inversion: Closing the Guidance Gap Advanced

Recall classifier-free guidance from Chapter 33: the guided noise prediction extrapolates from the unconditional (null) prediction toward the conditional one, $\epsilon_{\text{guided}} = \epsilon_\varnothing + w(\epsilon_c - \epsilon_\varnothing)$, where $\epsilon_\varnothing$ uses an empty "null-text" embedding. Null-text inversion makes a clever observation: the null embedding is just another input, and we are free to optimize it per timestep so that the guided generation trajectory lands back on the inverted (unguided) trajectory we recorded. Concretely, you first run DDIM inversion to get a reference trajectory $\{z_t^*\}$, then for each timestep optimize the null embedding $\varnothing_t$ to minimize the distance between the guided step's output and $z_{t-1}^*$:

Only the null embeddings are optimized; the model weights and the real prompt are fixed. After this short per-image optimization (a few hundred gradient steps over the timesteps, seconds to a minute on a GPU), the model can regenerate the real photo under full guidance with high fidelity, and any edit applied through the prompt now acts on a faithful reconstruction. Null-text inversion is what made convincing text-driven editing of arbitrary real photos practical, and it remains a strong baseline; lighter successors (negative-prompt inversion, which skips the optimization, and direct-inversion methods) trade a little fidelity for speed.

4. Prompt-to-Prompt: Editing Through Attention Advanced

With a faithful inversion, the question becomes how to apply an edit that changes the named thing and nothing else. Prompt-to-Prompt answers it through the cross-attention maps. Recall from Chapter 34 that cross-attention produces, for each word in the prompt, a spatial attention map showing where that word influences the image; the word "cat" attends to the cat's pixels. These maps carry the image's layout. Prompt-to-Prompt's insight is that to change content while keeping structure, you generate with the new prompt but inject the original prompt's attention maps for the words that are shared, and let only the changed word compute fresh attention.

So to turn "a photo of a cat" into "a photo of a dog," you keep the attention maps for "a photo of a" exactly as they were, which pins the composition, pose, and background, and let "dog" attend freely into the region the original "cat" occupied. The result is a dog in the cat's exact pose and scene. Three operations cover the common edits: map replacement swaps a word while reusing layout (cat to dog), map injection adds a word (adding "snowy") by re-using all original maps and inserting the new token's, and attention re-weighting strengthens or weakens a word's influence by scaling its map (more or less "snowy"). Figure 35.5.2 shows the swap.