The Vision Foundation Model Landscape

"Once there were a thousand small models, each expert in one narrow thing and helpless outside it. Now there are a few large ones that know almost everything a little, and the real skill is no longer training a model but choosing which giant to borrow and what to ask of it."

A Foundation Model Surveying Its Own Crowded Field

Your next vision project will almost certainly start with a downloaded backbone, not a trained one, and the single most consequential decision you make will be which giant to borrow. The previous five sections built the methods; this one turns them into the map you reach for when you face that choice. We will lay out the major foundation models by what they are good for, give a concrete decision procedure for choosing a backbone, examine the scaling laws and data-curation lessons that explain why these models work, and close on the predictive-representation direction that links this chapter to the rest of the book. By the end you will be able to look at a new task and name the right pretrained model, the adaptation recipe, and the reason for both. It completes the foundation-model story that Chapter 22 and the transfer learning of Chapter 21 set up, and hands off to Chapter 26.

1. The Major Families, by What They Do Intermediate

The methods of this chapter crystallized into a handful of model families, each defined by its pretraining objective and therefore by what its features are good at. Understanding the map means understanding which objective produced which strength. DINOv2 (self-distillation plus masked modeling, no language) gives general-purpose features that excel frozen, across classification, dense correspondence, depth, and segmentation. Its successor DINOv3 (Meta AI, 2025) scales the same recipe to a 7-billion-parameter ViT trained on 1.7 billion curated images. It adds a regularizer the authors call Gram anchoring, which keeps dense features sharp over long training, and reports state-of-the-art dense prediction with frozen weights. The CLIP and SigLIP family (language-supervised, from Section 25.4) gives language-aligned embeddings for anything involving text: zero-shot recognition, retrieval, and conditioning generators. The SAM line (promptable, from Section 25.5) gives masks on demand, and its latest member SAM 3 (Carion et al., 2025) adds promptable concept segmentation, returning masks for every instance matching a noun phrase in one model rather than one prompted object at a time. Table 25.6.1 lays out the comparison.

These families are increasingly combined rather than chosen exclusively. DINOv2 features feed monocular depth systems in Chapter 27; SigLIP encoders sit inside open-vocabulary detectors from Section 25.5 and inside multimodal language models; SAM consumes the boxes of a CLIP-based detector. The map is less a menu of alternatives than a toolbox of interoperable parts, which is the defining shift the chapter has been describing.

2. Choosing and Adapting a Backbone Beginner

For a real project, the choice follows from two questions answered in order. First, does your task involve language or an open vocabulary? If yes, start in the CLIP and SigLIP family, because only language-aligned models can match arbitrary text. Second, if the task is purely visual, will you freeze the backbone or fine-tune it? If you will freeze it (retrieval, clustering, few-shot, dense correspondence), choose DINOv2; if you will fine-tune on labeled data, an MAE-pretrained ViT is an excellent initialization. For masks of arbitrary objects, layer SAM on top of whichever recognition model names them. Figure 25.6.1 turns this into a decision tree.

Adapting the chosen backbone is cheap by design. The linear probe and fine-tuning protocols of Section 25.1 and Chapter 21 still apply, and parameter-efficient methods (a small adapter, or LoRA, the same low-rank adaptation you will meet for generators in Chapter 34) let you specialize a giant backbone by training a tiny fraction of its weights. The code below shows the most common adaptation: load a frozen DINOv2 backbone and attach a small trainable head.

import torch
import torch.nn as nn

# Load a frozen DINOv2 backbone; this is the practical default for frozen features.
backbone = torch.hub.load("facebookresearch/dinov2", "dinov2_vitb14")
for p in backbone.parameters():
    p.requires_grad_(False)                  # freeze: adapt with a tiny head only
backbone.eval()

class FoundationClassifier(nn.Module):
    def __init__(self, backbone, feat_dim=768, num_classes=10):
        super().__init__()
        self.backbone = backbone             # frozen feature extractor
        self.head = nn.Linear(feat_dim, num_classes)   # the only trainable parameters

    def forward(self, x):
        with torch.no_grad():
            feats = self.backbone(x)         # frozen forward, no gradients stored
        return self.head(feats)              # train just this linear layer

model = FoundationClassifier(backbone)
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
total = sum(p.numel() for p in model.parameters())
print(f"trainable: {trainable:,} of {total:,} params ({100*trainable/total:.3f}%)")
# trainable: 7,690 of 86,588,170 params (0.009%)

The printed line is the whole point of the chapter in one number: a task is solved by training under one ten-thousandth of the model's parameters, because the foundation model already learned to see. The practical example shows a team living by this workflow, and the Hands-On Lab at the end of this section has you build that workflow yourself, comparing a frozen DINOv2 backbone against CLIP zero-shot classification on the same images.

The illustration below sums up this shift: one shared giant backbone with small swappable adapter heads, retiring the old one-model-per-task era.

3. Scaling, Curation, and the Predictive Frontier Advanced

Why do these models work, and where are they going? Two empirical lessons explain the present. First, scaling laws: foundation-model quality improves predictably as a power law in model size, data size, and compute, so much of the progress from CLIP to DINOv2 to SigLIP came from scaling a known recipe rather than inventing a new one. Second, and the harder-won lesson, data curation matters as much as scale: DINOv2's leap over plain self-supervision came substantially from automatically curating a large, diverse, deduplicated image set, and the CLIP-data line (DataComp, DFN) showed that filtering web pairs well beats simply collecting more. Quality of data, not just quantity, sets the ceiling.

The frontier direction unifies the chapter's threads. Pixel reconstruction (MAE) wastes capacity on imperceptible detail; pure language alignment (CLIP) can miss fine spatial structure. The Joint-Embedding Predictive Architecture (JEPA) line argues for predicting in representation space: mask part of the input and predict the features of the missing part, not its pixels or its caption. I-JEPA did this for images, and the V-JEPA line extended it to video, learning to predict masked spatiotemporal regions in feature space and thereby learning motion and dynamics without action labels. V-JEPA 2 (Meta AI, 2025) scaled this to a 1.2-billion-parameter video model pretrained on over a million hours of video, and an action-conditioned variant plans robot manipulation from learned dynamics, an explicit step from representation learning toward a world model. This is the same self-supervision-to-prediction arc the cross-reference map traces: the masked-prediction idea of Section 25.3, pushed from reconstructing content to predicting representations, becomes a way to learn the structure of how scenes evolve. That is the doorway to the predictive video models and world models of Chapter 26 and beyond.

4. Generative Vision-Language Models and Visual Question Answering Intermediate

Everything so far in this chapter scores or clusters images; nothing in it can answer a question about one. CLIP (Section 25.4) can tell you that an image is more similar to the text "a child flying a kite" than to "a dog on a beach", but ask it "how many kites are in the sky?" and it has no machinery to reply, because it only measures similarity between a whole image and a whole caption. A foundation-models course that stops at CLIP leaves out the half of the multimodal field that actually talks: the models that read an image and generate free-form text about it, answering questions, describing scenes, and holding a dialogue. This subsection adds that half and ties it back to the contrastive story you already know.

4.1 The Task: Visual Question Answering

The cleanest way to test whether a model truly understands an image, rather than just labels it, is to ask it arbitrary questions: "What color is the bus?", "Is the man wearing a helmet?", "How many people are seated?" This is Visual Question Answering (VQA), and the problem it exposed is instructive. The first large benchmark let models cheat: questions carried such strong language priors that a model could answer "What color is the banana?" with "yellow" without looking at the image at all. The VQAv2 benchmark (Goyal et al., 2017, "Making the V in VQA Matter") fixed this by rebalancing: every question is paired with two similar images that yield different answers, so a model that ignores the pixels and bets on the prior is wrong half the time by construction. The "V" in VQA only matters if the vision is load-bearing, and VQAv2 forces it to be.

VQA also needs a metric that tolerates the genuine disagreement among human annotators (is that animal a "puppy" or a "dog"?). VQAv2 collects ten human answers per question and scores a candidate answer by how many humans gave it, saturating at three:

Read term by term: the numerator counts how many of the ten annotators produced the candidate answer $a$; dividing by three means an answer that at least three humans gave earns full credit, and the outer $\min$ caps the score at $1$ so extra agreement does not overflow. Concretely, if the model answers "yellow" and four of ten humans also said "yellow", the score is $\min(4/3, 1) = 1$; if only two humans said it, the score is $2/3 \approx 0.67$; if zero did, the score is $0$. The design rewards answers that match the human consensus while not punishing a model for missing a rare phrasing a single annotator happened to use. GQA (Hudson & Manning, CVPR 2019) extends VQA toward compositional reasoning, generating questions from scene graphs ("Is the cup to the left of the plate that is on the table?") so that answering correctly requires chaining several relational steps rather than recognizing one object.

The deeper shift is architectural. Early VQA systems framed the task as classification: encode the image into a feature vector, encode the question into an embedding, fuse the two, and predict over a fixed answer vocabulary of a few thousand frequent answers. That design cannot produce any answer outside its vocabulary and cannot explain itself. Modern vision-language models discard the fixed answer set entirely and treat VQA as open-ended autoregressive text generation: the model reads the image and the question and generates the answer one token at a time, exactly as a language model continues a sentence. Captioning, VQA, and multi-turn dialogue stop being three architectures and become one, differing only in the prompt.

4.2 The General Pattern: Encoder, Connector, LLM

Almost every modern vision-language model follows one template, and once you see it the individual models become variations on a theme. An image goes into a vision encoder (frequently a frozen CLIP or SigLIP ViT, reusing exactly the contrastively pretrained perception of Section 25.4), which emits a grid of patch features. Those features pass through a connector that turns them into visual tokens living in the language model's embedding space. Those visual tokens are then inserted into the context window alongside the embedded text tokens, and a language-model decoder autoregressively generates text conditioned on both. The image becomes, in effect, a few dozen "words" in a language the LLM already understands, and from there captioning, VQA, and dialogue are all just text continuation. Figure 25.6.2 shows the flow.

The models differ almost entirely in the connector, because that is where the design choices live: how many visual tokens, how much computation, and how much it can compress. Three families dominate. The Q-Former (BLIP-2) uses a small set of learned query tokens that cross-attend to the encoder's patches and compress them into a fixed, small number of visual tokens. The linear or MLP projector (LLaVA) is the minimalist choice: a single matrix multiply (or a two-layer MLP) maps each patch feature directly to a visual token, with almost no parameters and no compression. The perceiver resampler (Flamingo) resamples a variable number of patch features into a fixed number of latent tokens through cross-attention. The trade-off is the classic one between inductive bias and simplicity: the Q-Former and resampler bake in a learned compression step (more parameters, fewer visual tokens, more architectural assumptions), while the MLP projector trusts the LLM to do the heavy lifting (almost no parameters, one token per patch, minimal assumptions). LLaVA's result, that a trivial projector works remarkably well, was itself a finding.

4.3 BLIP-2: A Frozen Encoder, a Frozen LLM, and a Q-Former Between Them

Training a vision-language model from scratch means paying for both a vision backbone and a language model, an enormous cost. BLIP-2 (Li et al., 2023) asked whether you could leave both giants frozen and train only a small bridge between them. Its bridge is the Querying Transformer (Q-Former): a lightweight transformer holding a small set of learned query tokens (32 in the original) that are not derived from the image. These queries do two things at once. They cross-attend to the frozen image encoder's patch features, pulling in visual information, and they self-attend among themselves, organizing that information. The query outputs, projected to the language model's token dimension, are the entire visual input the frozen LLM ever sees: the image is compressed into just 32 tokens. Only the Q-Former trains; the vision encoder and the LLM stay frozen, which is why BLIP-2 reaches strong zero-shot VQA with far fewer trainable parameters than a model like Flamingo.

BLIP-2 trains the Q-Former in two stages. Stage one is representation learning: with the LLM not yet attached, the Q-Former is trained against the frozen image encoder using three objectives borrowed from contrastive and matching work, image-text contrastive learning, image-text matching, and image-grounded text generation, so the queries learn to extract text-relevant visual features. Stage two is generative learning: the trained queries are projected and prepended to the frozen LLM's context as a soft visual prefix, and the system learns to generate text conditioned on that prefix. The split matters because it first teaches the queries what to look at (stage one) before teaching them to speak to a language model (stage two).

4.4 LLaVA: Visual Instruction Tuning With a Simple Projector

BLIP-2 keeps the LLM frozen and invests in a clever connector; LLaVA (Liu et al., 2023, "Visual Instruction Tuning", NeurIPS 2023) makes the opposite bet. Its connector is as simple as possible, a single linear projection (a 2-layer MLP in LLaVA-1.5) mapping CLIP ViT patch features into the embedding space of a Vicuna language model, and its innovation is the data. The authors used a text-only GPT-4 to generate visual instruction-tuning data: given the captions and bounding boxes of an image (as text), GPT-4 wrote rich questions, answers, and conversations about it, producing instruction-following data for images without any human annotating new images. LLaVA then trains in two stages: stage one aligns the projector with the vision encoder and the LLM both frozen, so only the small projector learns to speak the LLM's language; stage two instruction-tunes the projector and the LLM together on the GPT-4-generated dialogues.

LLaVA-1.5 (Liu et al., 2023, "Improved Baselines with Visual Instruction Tuning", CVPR 2024) showed how far this simple recipe scales with a few disciplined upgrades: a stronger CLIP-ViT-L vision encoder at 336-pixel resolution (more visual detail), the two-layer MLP projector in place of the single linear layer, and the addition of academic VQA datasets to the instruction mix. The result reached state of the art on eleven benchmarks, and the 13-billion-parameter model trains in roughly one day on a single 8xA100 node, a striking efficiency for the capability. LLaVA's broader lesson is that a minimal connector plus good instruction data can rival far more elaborate architectures, which is why the linear-or-MLP projector became the default for a wave of open vision-language models.

4.5 The Wider Landscape and How These Models Are Evaluated

BLIP-2 and LLaVA anchor two ends of a fast-moving field. Flamingo (Alayrac et al., 2022) pioneered the frozen-giants approach and added few-shot multimodal learning over interleaved sequences of images and text, using a perceiver resampler and gated cross-attention layers spliced into a frozen language model. More recent open families push resolution, grounding, and data quality: the Qwen2-VL and Qwen2.5-VL line (Wang et al., 2024, arXiv:2409.12191; Bai et al., 2025, arXiv:2502.13923) handles dynamic input resolution and visual grounding; InternVL (Chen et al., 2023, arXiv:2312.14238) scales the vision encoder alongside the LLM; PaliGemma (Beyer et al., 2024, arXiv:2407.07726) pairs a SigLIP encoder with the Gemma language model; and Molmo, trained on the open PixMo data (Deitke et al., 2024, arXiv:2409.17146), emphasizes fully open training data. Proprietary multimodal assistants and open model-card releases (for example the Llama-3.2-Vision models, distributed as model-card releases rather than arXiv papers) round out the ecosystem.

Because these models generate open-ended text, evaluation has grown beyond the classic VQAv2 and GQA accuracy. MMMU (Yue et al., 2024, "MMMU: A Massive Multi-discipline Multimodal Understanding benchmark", arXiv:2311.16502) tests college-level reasoning across dozens of subjects with figures, charts, and diagrams; MMBench probes specific capabilities with a robust answer-extraction protocol; and MM-Vet evaluates integrated skills such as combining recognition, OCR, and spatial reasoning in a single answer. Together they measure whether a model genuinely reasons over an image or merely pattern-matches frequent question-answer pairs, the same concern that motivated VQAv2's rebalancing in the first place.

4.6 Code-First: Visual Question Answering in a Dozen Lines

The internals above (a frozen encoder, a connector, a frozen-or-tuned LLM, two-stage training) are exactly what a library now hides behind one class. Hugging Face transformers ships these models with a processor that handles image preprocessing and prompt formatting and a generation method that runs the autoregressive decode. The complexity of building the pipeline by hand collapses to loading a checkpoint and calling generate: the same image-plus-question to answer that took a research lab months to train is, for inference, a dozen lines.

import torch
from PIL import Image
from transformers import Blip2Processor, Blip2ForConditionalGeneration

# Load a pretrained BLIP-2: frozen ViT + Q-Former + frozen LLM, all in one checkpoint.
processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
model = Blip2ForConditionalGeneration.from_pretrained(
    "Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16
)
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

image = Image.open("street.jpg").convert("RGB")     # your own image
question = "Question: How many people are crossing the street? Answer:"

# The processor turns the image into pixel values and tokenizes the prompt;
# the Q-Former compresses the image to ~32 visual tokens fed to the frozen LLM.
inputs = processor(images=image, text=question, return_tensors="pt").to(
    device, torch.float16
)
generated = model.generate(**inputs, max_new_tokens=20)  # autoregressive decode
answer = processor.batch_decode(generated, skip_special_tokens=True)[0].strip()
print(answer)                                            # open-ended text, no fixed vocab

The payoff mirrors the frozen-backbone reveal of subsection 2: the entire encoder-connector-LLM machine, frozen giants and learned bridge alike, is downloaded as one checkpoint and driven with a processor and a generate call. What the library absorbs is substantial: image preprocessing to the encoder's expected resolution, insertion of the visual tokens at the right position in the prompt, the key-value caching that makes autoregressive decoding fast, and the half-precision memory management that lets a multi-billion-parameter model run on one GPU. Understanding the pipeline is what lets you choose between BLIP-2's compressing Q-Former and LLaVA's pass-through projector for your latency and accuracy budget; the library is what lets you run either in an afternoon.

pip install torch torchvision transformers pillow numpy

from pathlib import Path
from PIL import Image
import torch
from torchvision import transforms

paths = sorted(Path("images").glob("*.jpg")) + sorted(Path("images").glob("*.png"))
pil_images = [Image.open(p).convert("RGB") for p in paths]
print(f"loaded {len(pil_images)} images")

# DINOv2 expects a square crop sized to a multiple of the patch (14) and
# the standard ImageNet mean/std normalization.
# TODO: build a torchvision transform that Resize((224, 224)), ToTensor(),
# and Normalize(mean=[0.485,0.456,0.406], std=[0.229,0.224,0.225]).
dino_tf = ...

backbone = torch.hub.load("facebookresearch/dinov2", "dinov2_vitb14")
backbone.eval()
for p in backbone.parameters():
    p.requires_grad_(False)            # frozen feature extractor, no training

batch = torch.stack([dino_tf(im) for im in pil_images])
with torch.no_grad():
    feats = backbone(batch)            # (N, 768) CLS-token features
# TODO: L2-normalize feats along dim=1 so each row has unit length,
# then compute the (N, N) cosine-similarity matrix as feats @ feats.T.
dino_emb = ...
sim = ...

from transformers import CLIPModel, CLIPProcessor

clip = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
proc = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
clip.eval()

labels = ["a dog", "a cat", "a car", "a plate of food", "a building"]
# TODO: build prompt strings with a template like "a photo of {label}",
# call proc(text=prompts, images=pil_images, return_tensors="pt", padding=True),
# run clip(**inputs), then softmax outputs.logits_per_image over dim=1.
probs = ...

for i, path in enumerate(paths):
    top = probs[i].argmax().item()
    conf = probs[i][top].item()
    # TODO: get the indices of the 3 most similar images to image i,
    # excluding i itself, using sim[i] (set sim[i, i] to -1 first or skip it),
    # then print path.name, labels[top], conf, and the neighbor file names.
    ...

# TODO: scan for a pair (i, j) where probs[i].argmax() == probs[j].argmax()
# (same CLIP label) but sim[i, j] is low (DINOv2 thinks they look different),
# print both file names and the values, and note the disagreement in a comment.
...

# Label-free vision toolkit: DINOv2 nearest neighbors + CLIP zero-shot labels.
from pathlib import Path
from PIL import Image
import torch
import torch.nn.functional as F
from torchvision import transforms
from transformers import CLIPModel, CLIPProcessor

# --- Step 1: load and preprocess ---
paths = sorted(Path("images").glob("*.jpg")) + sorted(Path("images").glob("*.png"))
pil_images = [Image.open(p).convert("RGB") for p in paths]
print(f"loaded {len(pil_images)} images")

dino_tf = transforms.Compose([
    transforms.Resize((224, 224)),                       # 224 is a multiple of 14
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),     # ImageNet normalization
])

# --- Step 2: DINOv2 embeddings and cosine-similarity matrix ---
backbone = torch.hub.load("facebookresearch/dinov2", "dinov2_vitb14")
backbone.eval()
for p in backbone.parameters():
    p.requires_grad_(False)

batch = torch.stack([dino_tf(im) for im in pil_images])
with torch.no_grad():
    feats = backbone(batch)                              # (N, 768)
dino_emb = F.normalize(feats, dim=1)                     # unit-length rows
sim = dino_emb @ dino_emb.T                              # cosine similarity (N, N)

# --- Step 3: CLIP zero-shot classification ---
clip = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
proc = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
clip.eval()

labels = ["a dog", "a cat", "a car", "a plate of food", "a building"]
prompts = [f"a photo of {lbl}" for lbl in labels]       # prompt template helps
inputs = proc(text=prompts, images=pil_images, return_tensors="pt", padding=True)
with torch.no_grad():
    out = clip(**inputs)
probs = out.logits_per_image.softmax(dim=1)             # (N, num_labels)

# --- Step 4: per-image report ---
for i, path in enumerate(paths):
    top = probs[i].argmax().item()
    conf = probs[i][top].item()
    row = sim[i].clone()
    row[i] = -1                                          # exclude self
    nbrs = [paths[j].name for j in row.topk(3).indices.tolist()]
    print(f'{path.name} -> "{labels[top]}" ({conf:.2f}); neighbors: {", ".join(nbrs)}')

# --- Step 5: find a disagreement (same CLIP label, low DINOv2 similarity) ---
clip_label = probs.argmax(dim=1)
N = len(paths)
for i in range(N):
    for j in range(i + 1, N):
        if clip_label[i] == clip_label[j] and sim[i, j] < 0.4:
            print(f"DISAGREE: {paths[i].name} & {paths[j].name} both "
                  f'"{labels[clip_label[i]]}" but DINOv2 sim={sim[i, j]:.2f}')
            break
    else:
        continue
    break
# DINOv2 groups by appearance; CLIP groups by nameable concept. The gap between
# them is exactly the choose-by-objective lesson of Table 25.6.1.

Further Reading: Generative Vision-Language Models Intermediate