Generative Vision Models

Part Overview

Every part of this book so far has pointed the arrow in one direction: an image goes in, and meaning comes out. Part I conditioned the signal, Part II extracted geometry and structure from it, and Part III trained networks to recognize, detect, segment, and reconstruct. Part IV reverses the arrow. The models in these nine chapters take a text description, a sketch, a noise vector, or nothing at all, and produce the image itself. That reversal demands more than a clever architecture: a model can only generate convincing images if it has captured, in some form, the entire distribution of natural images rather than just the decision boundaries between classes.

The part opens by making that idea precise. Chapter 30 explains what it means to model p(x), maps the family tree of generative approaches, and introduces the vocabulary of latent spaces, sampling, and evaluation that every later chapter relies on. Chapter 31 then builds the first working generator out of a humble idea, compression: autoencoders become variational autoencoders, and discrete VQ codebooks lay groundwork the diffusion chapters will quietly reuse. Chapter 32 covers GANs, the adversarial family that first made photorealistic generation possible, along with the training pathologies and latent-space editing tricks that remain relevant long after the spotlight moved on. Chapter 33 arrives at diffusion, where the denoising problem of Chapter 7 returns in learned form: corrupt an image with noise, train a network to undo the damage, and the result is the engine behind essentially all modern image generation, including the guidance and latent-space variants that make it practical.

With the engine understood, the next three chapters put it to work. Chapter 34 assembles complete text-to-image systems, connecting the CLIP encoders of Chapter 25 to diffusion backbones and walking through Stable Diffusion, the wider model landscape, prompting, and fine-tuning. Chapter 35 turns prompt roulette into precise control: spatial conditioning with ControlNet, personalization with LoRA and DreamBooth, and editing methods that change one thing while preserving everything else, the generative descendant of Chapter 7's inpainting. Chapter 36 grows new axes entirely, extending generation into time, depth, and agency through video diffusion, text-to-3D, and world models that learn to simulate, reconnecting with the video and 3D vision of Chapters 26 and 27.

The final chapters close the loop. Chapter 37 confronts the questions generation raises: how to measure what a generator produces, how to govern deepfakes, watermarking, and licensing, and how to harness generators as synthetic-data engines that train the recognition models of Part III. This is where the book's two halves meet: the models that create images feeding the models that understand them. Chapter 38 ends the part with a consolidated tools reference for the whole generative stack.

Chapters: 9 (Chapters 30–38). Read Chapters 30 through 33 in order; they build the theory each successive chapter assumes. The applied chapters that follow can be read selectively, though each leans on the diffusion machinery of Chapter 33.