The Model Landscape: DALL-E, Imagen, Midjourney & FLUX

"We are not rivals. We are the same recipe arguing about which encoder to hire and how much to retouch the photos afterward. The benchmark leaderboard is just our group chat, made public."

A Family of Text-to-Image Models at an Awkward Reunion

The previous two sections built the template: an encoder (Section 34.1) produces conditioning, a denoiser (Section 34.2) iterates in a latent, an autoencoder decodes. This section treats the template as fixed and varies the parts. The goal is not to memorize a feature list, which is stale within months, but to learn the axes along which systems differ so that a model released after this book is printed still slots into the same mental map.

1. The Three Knobs Beginner

Here is the surprise that makes the whole landscape tractable: nearly every visible difference between these systems, the prompt following, the signature look, the step count, traces to just three design choices. Learn the three and you can predict a system's strengths before you run it.

• Encoder. CLIP-only (tight visual grounding, weak compositionality), large language-model encoder like T5 (strong prompt following), or both. This is the single biggest driver of prompt-following quality, the encoder-ceiling argument of Section 34.1.
• Generator path. A two-stage prior-plus-decoder (unCLIP), a single conditional latent-diffusion U-Net, or a diffusion transformer trained with flow matching. (Rectified flow, which appears throughout this section, is the flow-matching variant from Chapter 33 that learns a near-straight path from noise to image, so generation needs fewer steps.) This drives sample quality, scaling behavior, and step count.
• Data and finishing. What captions the model trained on, what aesthetic filtering shaped the training set, and what post-processing (refinement, upscaling, safety filtering) runs at inference. This drives the intangible "look" and the prompt-vocabulary that works.

Table 34.3.1 places the major systems on these axes. Read it as a coordinate map, not a ranking: each system occupies a position chosen for a purpose.

2. DALL-E: Two Very Different Systems Under One Name Intermediate

DALL-E 2 and DALL-E 3 share a brand and almost nothing else, which is instructive. DALL-E 2 is the canonical unCLIP design: a prior network maps the CLIP text embedding to a CLIP image embedding, and a diffusion decoder generates an image from that image embedding. The two-stage split exploits CLIP's shared space directly (Section 34.1): generating in CLIP image-embedding space first, then decoding, was meant to give a clean handle on semantic content. In practice the prior added complexity and the binding problems of CLIP propagated through it.

DALL-E 3 abandoned that story and made a data argument instead. Its central finding is that prompt following is bottlenecked by training-caption quality: web alt-text is short and noisy, so a model trained on it learns sloppy text-image correspondence. DALL-E 3 trained a captioner to produce long, detailed, accurate captions for the training images, then trained the generator on those. The result follows complex prompts dramatically better, with no exotic architecture; the diffusion backbone is conventional. This is the clearest case in the landscape of the third knob (data and captions) dominating the first two.

3. Imagen and the Encoder Bet Intermediate

Imagen made the encoder argument concrete and won it. It used a frozen T5-XXL text encoder, never fine-tuned on images, and a cascade of pixel-space diffusion models: a base model at low resolution followed by two super-resolution diffusion stages (the diffusion upscaler thread from Chapter 33, itself a learned descendant of the classical super-resolution of Chapter 7). The headline result was that scaling the frozen text encoder improved image-text alignment more than scaling the diffusion model did, the empirical foundation for the T5 adoption you see in SD3 and FLUX. Imagen ran in pixel space rather than latent space, which made it expensive but sidestepped the VAE; later systems mostly chose the latent path for cost.

4. The Open Line: Stable Diffusion to FLUX Intermediate

The open-weight lineage is where most practitioners actually work, because the weights are downloadable and the fine-tuning ecosystem of Section 34.6 is built around it. Stable Diffusion 1.5 is the canonical latent-diffusion U-Net with a single CLIP encoder. SDXL scaled it up, added a second CLIP text encoder for richer conditioning, and introduced an optional refiner model that runs a few extra denoising steps to sharpen details. SD3, and its SD3.5 refresh (Stability AI, October 2024), made the generational jump: they replaced the U-Net with the MMDiT transformer of Section 34.2, trained with the rectified flow of Chapter 33, and conditioned on three encoders (two CLIP plus T5).

FLUX.1 (Black Forest Labs, August 2024), built by founders who had worked on the original Stable Diffusion, is a large rectified-flow transformer that set the open-weight bar on release. It repaired the two failure modes that long plagued the field: rendering legible text within images and producing correct hands. The open frontier has kept moving since: FLUX.2 (November 2025) pairs the rectified-flow transformer with a vision-language encoder, and competitive open models such as Qwen-Image and HiDream-I1 now trade the top spots on public arenas, so any single "strongest open model" claim is short-lived. The durable point is that they all sit at the same coordinates in Table 34.3.1. The illustration below makes the family resemblance literal: identical bodies, differently set dials.

You can load and run several of these from one library, which is the practical payoff of the shared template. The code below runs SDXL and SD3 through nearly identical calls.

import torch
from diffusers import StableDiffusionXLPipeline, StableDiffusion3Pipeline

# Run two generations of the same template through near-identical calls:
# SDXL (CLIP-only) and SD3 (CLIP + T5). The "OPEN" sign in the prompt is a
# deliberate stress test of the encoder difference between them.
prompt = ("a vintage bookstore at dusk, warm lamplight, a black cat on the "
          "counter, a sign reading OPEN in the window")

# SDXL: latent-diffusion U-Net with two CLIP encoders.
sdxl = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16).to("cuda")
img_sdxl = sdxl(prompt, num_inference_steps=30, guidance_scale=7.0).images[0]

# SD3: rectified-flow MMDiT with two CLIP encoders plus T5.
sd3 = StableDiffusion3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium-diffusers",
    torch_dtype=torch.float16).to("cuda")
img_sd3 = sd3(prompt, num_inference_steps=28, guidance_scale=7.0).images[0]

# The legible-sign prompt exposes the encoder gap: SD3's T5 reads "OPEN".
img_sdxl.save("bookstore_sdxl.png")
img_sd3.save("bookstore_sd3.png")

That single code listing demonstrates the chapter's thesis: swapping a frontier system for its predecessor is a two-line change because both obey the template. The visible quality difference traces directly to the knobs in Table 34.3.1, not to incomparable architectures.

from diffusers import AutoPipelineForText2Image
import torch

# AutoPipeline reads the checkpoint's config and picks the right pipeline
# class, so the same three lines load SD 1.5, SDXL, SD3, or FLUX. Here it
# resolves FLUX.1-schnell, a distilled few-step model.
pipe = AutoPipelineForText2Image.from_pretrained(
    "black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16).to("cuda")
image = pipe("a black cat on a bookstore counter at dusk",
             num_inference_steps=4).images[0]   # schnell is a few-step model

5. Midjourney and the Aesthetic Knob Beginner

Midjourney is closed and undocumented, which makes it the perfect illustration of the third knob in isolation. Its architecture is presumably some diffusion variant, but its defining characteristic is the data and finishing: it is tuned hard toward a particular dramatic, painterly aesthetic, so even a bland prompt returns a polished, stylized image. This is a product decision, not an architectural one. The cost is control: the same aesthetic tuning that makes casual prompts look good fights you when you want a plain, literal, or documentary image. Understanding that Midjourney's "look" lives in the data-and-finishing knob, not in some secret architecture, tells you immediately why it excels at evocative concept art and struggles with neutral product photography, and why an open model with a neutral default is sometimes the better tool.