CLIP: Language as Supervision

"They never told me the names of things. They just showed me four hundred million photos and the captions people had already typed underneath them. By the time they were done, I could recognize a thing I had never been trained on, simply by reading its description. Turns out the internet is one enormous, badly punctuated labeled dataset."

A CLIP Encoder Fluent in Pictures and Words

Every method so far in this chapter learned from pixels alone. CLIP (Radford et al., 2021) brings in the one signal the internet produces at planetary scale for free: the text humans write alongside images. We will see how an image and its caption are encoded into a common space, write the symmetric contrastive loss that aligns them, build a zero-shot classifier out of nothing but text prompts, and understand why four hundred million noisy web pairs produced more robust features than a million clean ImageNet labels. This is the section where the learned descriptor of Chapter 10 finally becomes a universal one, comparable not just to other images but to words, and it is the exact component that will let a text prompt drive an image generator in Chapter 34.

1. Two Encoders, One Embedding Space Intermediate

CLIP has two networks. An image encoder (a ViT from Chapter 22, or a ResNet from Chapter 20) maps an image to a vector. A text encoder (a transformer) maps a caption to a vector of the same dimension. Both are followed by a linear projection into a shared $d$-dimensional space, and both outputs are L2-normalized so they lie on the unit hypersphere, where the dot product is the cosine similarity. Once each vector has length one, the $\|u\|\|v\|$ denominator in the cosine formula from Section 25.2 equals one, so the plain dot product already is the cosine. The training data is a batch of $N$ image-caption pairs. The goal: each image should be closest to its own caption and far from the other $N - 1$ captions, and symmetrically each caption should be closest to its own image. Figure 25.4.1 shows the contrastive matrix this sets up.

The loss is the InfoNCE of Section 25.2 applied in both directions. Let $I_i$ be the normalized embedding of image $i$, $T_j$ the embedding of caption $j$, and $\tau$ a learned temperature. The image-to-text loss treats each image's matching caption as the correct class among all $N$ captions; the text-to-image loss does the reverse. The total is their average,

Both terms are just cross-entropy over the rows and columns of the similarity matrix, with the correct class on the diagonal. The implementation is strikingly short for what it produces.

import torch
import torch.nn.functional as F

def clip_loss(image_emb, text_emb, logit_scale):
    """image_emb, text_emb: (N, d). logit_scale = exp(learned temperature). Symmetric."""
    image_emb = F.normalize(image_emb, dim=1)        # onto the unit hypersphere
    text_emb = F.normalize(text_emb, dim=1)
    logits = logit_scale * image_emb @ text_emb.t()  # (N, N) similarity matrix
    targets = torch.arange(image_emb.size(0), device=image_emb.device)  # diagonal = matches
    loss_i = F.cross_entropy(logits, targets)        # each image picks its caption (rows)
    loss_t = F.cross_entropy(logits.t(), targets)    # each caption picks its image (cols)
    return (loss_i + loss_t) / 2

torch.manual_seed(0)
img = torch.randn(6, 512); txt = torch.randn(6, 512)
scale = torch.tensor(100.0)                          # exp of the learned temperature
print("CLIP loss:", round(clip_loss(img, txt, scale).item(), 4))
# CLIP loss: ~ high for random embeddings; drops toward 0 as the diagonal dominates

That is the entire training objective. There is no classification head, no fixed label set, no taxonomy. The supervision is purely "this caption goes with this image and not the others", repeated over four hundred million pairs. The temperature is learned (parameterized as $\exp$ of a free scalar so it stays positive) and clamped to avoid runaway scaling. Everything CLIP can do downstream emerges from this single alignment.

The illustration below pictures this shared space as a town square where each photo finds and stands beside its matching caption.

2. Zero-Shot Classification: A Prompt Becomes a Classifier Intermediate

Here is the payoff that startled the field. CLIP was never trained to classify ImageNet, yet it classifies ImageNet competitively without seeing a single labeled ImageNet example. The trick: turn each class name into a sentence, embed all the sentences with the text encoder, embed the image with the image encoder, and pick the class whose sentence embedding is most similar to the image. The class names are the only thing you supply; no training, no labeled examples, no fine-tuning. A new set of classes is a new set of sentences, computed in seconds. This is what zero-shot means: the model generalizes to categories specified only at inference time.

The sentence template matters more than it should, an effect called prompt engineering. The bare class name "cat" embeds less usefully than "a photo of a cat", because the training captions were natural sentences, so a sentence-shaped prompt sits closer to the image distribution the text encoder learned. Averaging the embeddings of several templates ("a photo of a {}", "a blurry photo of a {}", "a {} in the wild") gives a more robust class vector still. The code below builds a zero-shot classifier from class names alone.

import torch
import torch.nn.functional as F

def zero_shot_classifier(class_names, text_encoder, tokenizer, templates):
    """Build a (num_classes, d) weight matrix from text prompts, no training at all."""
    weights = []
    for name in class_names:
        prompts = [t.format(name) for t in templates]          # "a photo of a cat", ...
        tokens = tokenizer(prompts)
        embs = F.normalize(text_encoder(tokens), dim=1)        # (num_templates, d)
        weights.append(F.normalize(embs.mean(0), dim=0))       # average then renormalize
    return torch.stack(weights)                                # (num_classes, d)

def classify(image_emb, class_weights):
    """Pick the class whose text embedding is most similar to the image embedding."""
    image_emb = F.normalize(image_emb, dim=1)
    logits = image_emb @ class_weights.t()                     # (batch, num_classes)
    return logits.argmax(dim=1)

# Usage sketch (encoders from a pretrained CLIP):
templates = ["a photo of a {}.", "a blurry photo of a {}.", "a {} in nature."]
# W = zero_shot_classifier(["cat", "dog", "car"], text_encoder, tokenizer, templates)
# preds = classify(image_features, W)
print("a zero-shot classifier is just a stack of text embeddings; classes set at inference")

Because the classifier is just text, CLIP can recognize categories that no labeled dataset ever covered: a specific dog breed, a brand of car, an abstract attribute like "a person smiling". This open vocabulary is what carries forward into the open-vocabulary detection and segmentation of Section 25.5. The same property also makes CLIP a measurement tool: the similarity between a generated image and a text prompt, called CLIPScore, is a standard way to evaluate text-to-image models in Chapter 37.

3. Why Web Text Beat Clean Labels Advanced

CLIP's web captions are noisy: misspelled, irrelevant, sometimes wrong. A clean dataset like ImageNet has a million carefully verified labels. Intuition says clean labels should win, yet CLIP's features transferred better and, crucially, were far more robust to distribution shift, holding accuracy on stylized, sketched, and adversarially-collected versions of ImageNet where supervised models collapsed. Two reasons explain the surprise. First, scale: four hundred million pairs is four hundred times ImageNet, and the diversity of web images covers far more of the visual world than any curated set. Second, the supervision is richer per example: a free-form caption ("a brown dog catching a frisbee on a beach at sunset") carries far more information than a single class index, describing objects, attributes, relations, and context all at once. The model that must match that sentence learns a more compositional representation than the model that must emit one of a thousand integers.

This reframed a belief that had held since the start of deep learning. The bottleneck was never the absence of clean labels; it was the absence of scale, and the web provides scale in the form of weak, free, abundant language supervision. The transfer-learning lesson of Chapter 21 reaches its largest form here: pretrain on everything, then specialize cheaply. The practical example shows a team acting on exactly this.

# Zero-shot classification end to end with a pretrained CLIP:
# encode the image and the candidate prompts, then softmax their similarities.
import open_clip, torch
model, _, preprocess = open_clip.create_model_and_transforms("ViT-B-32", pretrained="laion2b_s34b_b79k")
tokenizer = open_clip.get_tokenizer("ViT-B-32")
image = preprocess(some_pil_image).unsqueeze(0)
text = tokenizer(["a photo of a cat", "a photo of a dog"])
with torch.no_grad():
    img_f = model.encode_image(image); txt_f = model.encode_text(text)
    probs = (img_f @ txt_f.T).softmax(dim=-1)        # zero-shot class probabilities