Section 32.5: GAN Inversion & Latent-Space Editing

"You handed me a photograph of a stranger and asked which of my dreams it came from. None of them, exactly. But I found a dream close enough that you cannot tell, and then I made her smile by walking three steps north in dream-space."

A Generator Asked to Run in Reverse

The latent arithmetic of DCGAN in Section 32.3 hinted that the latent space has semantic structure: directions correspond to meaningful changes. StyleGAN's disentangled $W$ space of Section 32.3 made that structure far cleaner. But all of it operated on generated images, where you already have the code. To edit a real photograph, the one your user actually uploaded, you first have to answer a hard question: which latent code, fed to the generator, produces this exact image? That is GAN inversion, and it is the bridge from "a GAN that dreams up faces" to "a tool that edits your face".

1. The Inversion Problem Intermediate

Formally, given a fixed pretrained generator $G$ and a target image $\mathbf{x}$, inversion seeks a latent code $\mathbf{w}^{*}$ such that $G(\mathbf{w}^{*}) \approx \mathbf{x}$. The "approximately equal" is measured with a reconstruction loss that combines a pixel term and a perceptual term (the Learned Perceptual Image Patch Similarity, or LPIPS, distance, which compares deep features rather than raw pixels and is far closer to human judgment, a metric developed in Chapter 37):

$$ \mathbf{w}^{*} \;=\; \arg\min_{\mathbf{w}} \; \lVert G(\mathbf{w}) - \mathbf{x} \rVert_2^2 \;+\; \lambda_{\text{lpips}} \, \mathrm{LPIPS}\big(G(\mathbf{w}), \mathbf{x}\big). $$

There are two ways to solve this, and they trade speed against quality.

Optimization-based inversion treats $\mathbf{w}$ as the variable and runs gradient descent on the loss above, with $G$ frozen. It is accurate but slow, hundreds to thousands of optimization steps per image, because every step is a full forward and backward pass through the generator. Encoder-based inversion trains a separate feed-forward network $E$ that maps an image directly to its latent in one pass, amortizing the optimization across a training set exactly as the VAE encoder of Chapter 31 amortized inference. The pixel2style2pixel (pSp) encoder (Richardson et al., 2021) is the canonical example: a single forward pass yields a $W+$ code, milliseconds instead of minutes. The cost is accuracy, the encoder generalizes but rarely matches per-image optimization, so the best systems use the encoder for an instant initialization and a few optimization steps to refine.

# Optimization-based GAN inversion: with the generator frozen, run gradient
# descent on the latent w alone so that G(w) reconstructs a target image,
# scoring the match with a pixel term plus the LPIPS perceptual distance.
import torch
import lpips   # pip install lpips; the standard perceptual metric

def invert_image(G, target, w_init, steps=500, lr=0.05, device="cuda"):
    """Optimization-based inversion: find w so that G(w) reconstructs `target`."""
    w = w_init.clone().detach().requires_grad_(True)   # optimize the latent, not G
    opt = torch.optim.Adam([w], lr=lr)
    perceptual = lpips.LPIPS(net="vgg").to(device)
    for step in range(steps):
        img = G.synthesis(w)                            # frozen generator forward pass
        loss = (img - target).pow(2).mean() + perceptual(img, target).mean()
        opt.zero_grad(); loss.backward(); opt.step()    # gradient flows only into w
    return w.detach()

2. Which Space? Z, W, and W+ Advanced

For a StyleGAN, you can invert into three different spaces, and the choice dominates the result. The original latent space $Z$ is the Gaussian input; inverting into it is hard and reconstructs poorly because $Z$ is entangled and low-capacity. The intermediate $W$ space (after the mapping network) is more disentangled and reconstructs noticeably better. The trick that makes inversion practical is $W+$: StyleGAN feeds the same $\mathbf{w}$ to every layer, but $W+$ relaxes this, allowing a different $\mathbf{w}$ per layer (eighteen vectors for a $1024 \times 1024$ generator). The extra degrees of freedom let $W+$ reconstruct almost any real image, including faces outside the training distribution, which $W$ alone cannot represent.

3. Finding Semantic Directions

Once you can place a real image in latent space, editing reduces to finding the right direction to move. There are three families of methods, and it helps to take them one at a time. Supervised directions use attribute labels: train a linear classifier (for example "smiling versus not") on many latent-image pairs, and its normal vector is the editing direction, walk along it to add a smile.

Unsupervised directions need no labels. GANSpace (Härkönen et al., 2020) runs principal component analysis (PCA) on a large sample of latent codes and finds that the top principal components correspond to interpretable changes (pose, lighting, gender), while SeFa factorizes the generator's first weight matrix to find directions in closed form. Conditional methods like StyleFlow round out the set, learning nonlinear edits that respect attribute correlations.

The supervised recipe is short enough to read in full. Collect a sample of latent codes, generate each image, label it with an off-the-shelf attribute classifier, fit a linear separator in latent space, and use its weight vector as the edit.

# Supervised editing direction: fit a linear SVM that separates "smiling"
# from "not smiling" latent codes in W space; its normal vector is the
# direction to walk for more or less of the attribute.
import numpy as np
from sklearn.svm import LinearSVC

def smile_direction(latents, attribute_scores):
    """latents: (N, w_dim) sampled W codes; attribute_scores: (N,) smiling probability.
       Returns a unit vector in W space that increases the attribute when added."""
    labels = (attribute_scores > 0.5).astype(int)          # binarize the attribute
    svm = LinearSVC(C=1.0).fit(latents, labels)            # separate smiling from not
    direction = svm.coef_[0]
    return direction / np.linalg.norm(direction)           # unit editing direction

# editing a real image: invert it, then walk along the direction
# w_edited = w_inverted + alpha * torch.tensor(smile_dir)   # alpha controls strength
# G.synthesis(w_edited) now shows the same person, smiling

The remarkable fact, and the reason this works at all, is that these directions are largely global and linear: the same "age" vector ages most faces, the same "pose" vector rotates most heads, because StyleGAN's $W$ space organized the factors of variation into roughly linear, disentangled axes. This is the editability that the entangled $Z$ space and, partly, the over-flexible $W+$ space lack. Figure 32.5.2 assembles the three steps into the single find-then-edit pipeline this section is built around.

Exercises