Personalization: LoRA, DreamBooth & Textual Inversion

"You showed me five photos of your dog and now I cannot stop drawing him. In a spacesuit. On the moon. I regret nothing."

A Diffusion Model That Has Bonded With Your Pet

Section 35.1 fixed where content goes by feeding the model a structural map; this section turns to the second axis of control, what a subject or style is, which no edge map can specify. Section 34.6 introduced these three methods at the level of "here is what each one does." This section takes them seriously as engineering choices. We work through what each one actually modifies, the loss that keeps DreamBooth from collapsing, the low-rank math that makes LoRA cheap, and a concrete comparison so that when a task lands on your desk you know which to reach for. We close with the production reality that you rarely use one adapter alone: you stack several LoRAs and weight them, which raises its own questions.

1. The Spectrum of Personalization Beginner

Picture the text-to-image stack from Chapter 34 as a line from input to output: text gets embedded, the embeddings condition the U-Net through cross-attention, and the U-Net's weights transform noise into a latent. A new concept can be injected at any of these stages. Inject it at the embedding (add a new word) and you have textual inversion. Inject it into the weights everywhere (fine-tune the U-Net) and you have DreamBooth. Inject it as small additive corrections inside the attention weights and you have LoRA. The illustration below dramatizes the escalating effort, and Figure 35.2.1 places the three on that line.

2. Textual Inversion: Learning One Word Intermediate

Textual inversion asks the smallest possible question: can a new concept be captured by a single new entry in the embedding table? It freezes the entire model and introduces one new token, written here as <my-cat>, whose embedding vector $v_*$ is the only thing that gets trained. Given a few images of the concept, you run ordinary diffusion training (corrupt the image, predict the noise) on prompts like "a photo of <my-cat>," and backpropagate only into $v_*$. The model's weights never move; you are searching the embedding space for the vector that, when fed through the frozen cross-attention, reconstructs your concept.

where $\theta$ is held fixed and $c(\cdot)$ is the frozen text encoder. The learned artifact is tiny: a single vector of a few thousand floats, often under 10 KB. That is also its limitation. One vector cannot express everything about a complex subject, so textual inversion captures broad style and rough identity well but struggles with fine, exact detail (the precise pattern on a face, the exact logo on a product). It is the method to reach for when you want a shareable, model-agnostic "word" for a look, and the model already knows the rough class your concept belongs to.

3. DreamBooth: Binding a Subject Without Forgetting Intermediate

DreamBooth goes to the other extreme and fine-tunes the whole U-Net so that a rare token (such as sks) becomes a faithful handle on a specific subject. Fine-tuning a giant model on three to five images invites two failures. The first is overfitting: the model memorizes the exact training shots and can only reproduce those poses. The second, more insidious, is language drift or catastrophic forgetting: as the model learns that sks dog means your dog, it starts forgetting what dogs in general look like, and prompts for any dog begin returning your dog.

DreamBooth's fix is the prior-preservation loss. Alongside the few subject images, you generate a few hundred images of the generic class ("a photo of a dog") using the model itself, and train on both sets at once. The subject term teaches the rare token; the class term reminds the model what the broader class looks like, anchoring the prior. The combined objective is

where the second term uses model-generated class images $x'$ with conditioning $c_{\text{class}} = $ "a photo of a dog," and $\lambda$ (often around $1.0$) balances the two. The result is a model that renders your subject in any pose, lighting, and context the prompt asks for, while still being able to draw generic members of the class. The price is size: a full fine-tuned U-Net is gigabytes, and you have one per subject unless you convert it to a LoRA.

4. LoRA: Low-Rank Adapters, the Modern Default Intermediate

LoRA sits in the productive middle. It comes from the observation, first made for language models in the transfer-learning setting of Chapter 21, that the change a fine-tune makes to a weight matrix is usually low rank: you do not need a full dense update, just a thin one. For a frozen weight matrix $W_0 \in \mathbb{R}^{d \times k}$, LoRA learns a low-rank correction $\Delta W = B A$ where $A \in \mathbb{R}^{r \times k}$ and $B \in \mathbb{R}^{d \times r}$ with rank $r \ll \min(d, k)$, and uses

Only $A$ and $B$ are trained; $W_0$ stays frozen. With $r$ as small as 4 to 16, the adapter has orders of magnitude fewer parameters than the full matrix, so the trained file is a few megabytes rather than gigabytes. $B$ is initialized to zero (the same start-at-identity trick as ControlNet's zero convolutions in Section 35.1), so the model is unchanged at the start of training. The scalar $\alpha / r$ sets the adapter's strength, and dividing by $r$ is what lets you change the rank without re-tuning everything else. A larger $r$ means $BA$ sums more terms, which would otherwise grow its magnitude; dividing by $r$ keeps the correction's scale roughly constant as you adjust capacity. In diffusion models, LoRA is applied to the attention projection matrices, the same Q, K, V, O matrices you met in the attention layers of Chapter 22. The code trains and then loads a LoRA.

# Load a trained subject LoRA into a frozen base model and render with it.
# The LoRA is a tiny file of low-rank A/B matrices; loading patches the
# attention projections, set_adapters scales each adapter's contribution.
import torch
from diffusers import StableDiffusionPipeline

# --- Inference: load a trained subject LoRA into a base model. ---
pipe = StableDiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")

# A LoRA is a small file of A/B matrices; loading it patches the attention weights.
pipe.load_lora_weights("my_dog_lora.safetensors", adapter_name="dog")
pipe.set_adapters(["dog"], adapter_weights=[0.8])   # inference-time strength multiplier

image = pipe(
    "a photo of sks dog wearing a tiny astronaut helmet on the moon",
    num_inference_steps=30,
).images[0]
image.save("astro_dog.png")

Because a LoRA is just a small additive delta to specific matrices, it has two properties that make it the default. It is shareable: a few-megabyte file rather than a multi-gigabyte checkpoint. And it is composable: you can load several at once and blend them, which subsection 6 covers. The training script (diffusers ships train_dreambooth_lora.py) combines DreamBooth's prior-preservation idea with LoRA's low-rank parameterization, giving subject fidelity at a fraction of the storage.

5. Choosing Among the Three Intermediate

The three methods are not competitors so much as points on a budget curve. Table 35.2.1 lays out the tradeoffs that decide which to use.

The rule of thumb: start with LoRA. It gives most of DreamBooth's fidelity at a thousandth of the storage and composes with other adapters. Reach for full DreamBooth only when you need the absolute best fidelity for one important subject and storage does not matter. Use textual inversion when you want a portable concept word that works across many base models, or when you only need to capture a style rather than an exact subject. All three are referenced in Chapter 34; the practical difference is almost always storage and composability, which is why the field converged on LoRA.

6. Stacking and Weighting Multiple LoRAs Advanced

The composability of LoRA is its quiet superpower and its quiet trap. Because each LoRA is an additive delta to the same matrices, you can load several at once: a subject LoRA for your character, a style LoRA for a watercolor look, and a detail LoRA for sharper textures. diffusers exposes this directly.

# Load two LoRAs at once (a subject and a style) and blend them with
# independent weights. Both add low-rank deltas into the same attention
# matrices, so their effects superpose and can also interfere.
pipe.load_lora_weights("character.safetensors", adapter_name="char")
pipe.load_lora_weights("watercolor.safetensors", adapter_name="style")

# Blend two adapters with per-adapter weights; they add into the same matrices.
pipe.set_adapters(["char", "style"], adapter_weights=[0.9, 0.6])

image = pipe("a portrait of sks person, painterly", num_inference_steps=30).images[0]

The trap is interference. Two LoRAs that touch the same matrices can fight: a strong style LoRA can wash out a subject's identity, or two subject LoRAs can blend into a chimera. The weights are the first line of defense, but for serious multi-concept work the field has developed merging methods (such as orthogonal or SVD-based merges) that combine adapters with less conflict than naive addition. The practical discipline is to tune one LoRA at a time, then add the next at a low weight and raise it until it just takes effect.