Open-Vocabulary Detection & Segmentation

"The old detector knew exactly eighty things and was magnificently helpless about the eighty-first. The new one does not have a list. You name the thing, in whatever words you like, and it goes and finds it. I find this both liberating and faintly terrifying."

A Detector That Lost Its Fixed Category List

In Section 25.4 CLIP turned a text prompt into an image classifier. This section carries that idea into the dense tasks of Chapter 23 and Chapter 24. We will see how a detector aligns each candidate region to text embeddings so its vocabulary becomes open, how segmentation follows the same recipe at the pixel level, and how the Segment Anything Model reframes segmentation as a promptable primitive trained on a billion masks. We will then assemble the grounded detect-then-segment pipeline that has become a default tool. The thread that began with hand-drawn boxes and masks ends with a system you steer by typing a phrase, and it sets up the foundation-model landscape of Section 25.6.

1. From a Fixed Head to Text Embeddings Intermediate

Recall how a closed-vocabulary detector classifies a region in Chapter 23: a region feature vector is multiplied by a fixed classification weight matrix, one column per known class, and a softmax picks the class. The matrix has exactly $C$ columns, so the detector can only ever name $C$ things; the eighty-first category does not exist because there is no eighty-first column. The open-vocabulary move is to replace that fixed weight matrix with text embeddings. Encode each candidate category name with a CLIP text encoder, and use those text vectors as the classification weights. Now adding a category means adding a sentence, not retraining a head, exactly the zero-shot trick of Section 25.4 moved from whole images to regions.

Concretely, if a region produces a feature $r \in \mathbb{R}^d$ projected into the CLIP space and the candidate categories have text embeddings $t_1, \ldots, t_K$, the region's class scores are the similarities

and the region is assigned the highest-scoring category (or labeled background if no score clears a threshold). The number of categories $K$ is whatever you supply at inference time. Figure 25.5.1 contrasts the closed head with the open one.

Models like ViLD, RegionCLIP, OWL-ViT, and Grounding DINO implement variants of this region-text alignment, differing mainly in how they obtain good region features and how they train the alignment (distilling from CLIP, or training on grounded image-text data that ties phrases to boxes). The detection mechanics, region proposals, the set-prediction transformer decoder, are the ones you built in Chapter 23; only the classification step changed.

2. Open-Vocabulary and Promptable Segmentation Intermediate

Segmentation gets the same treatment, at the pixel level. Language-driven segmenters (LSeg, and the open-vocabulary mask transformers that followed) compute a per-pixel or per-mask embedding in the CLIP space and label each by similarity to text queries, so you can segment "the wooden chair" without that class ever appearing in a training mask. The mask-transformer machinery of Chapter 24 supplies the masks; the text alignment supplies the open vocabulary.

A different and hugely influential idea arrived with the Segment Anything Model (SAM, Kirillov et al., 2023). SAM does not classify at all. It is a promptable segmenter: given an image and a prompt (a click point, a box, or a rough mask), it outputs a high-quality mask for the object indicated by the prompt. It was trained on SA-1B, a dataset of 1.1 billion masks built with a data engine in which the model itself proposed masks that annotators refined, bootstrapping to a scale no manual effort could reach. Because the prompt rather than a label specifies the target, SAM segments objects it was never told the names of; it is a segmentation foundation model, a reusable primitive rather than a task-specific network. Figure 25.5.2 shows its three-part architecture.

SAM's design choice that makes it usable interactively is the split: the expensive image encoder runs once, and every subsequent prompt only invokes the cheap decoder, so a user can click around an image and get instant masks. The code below shows the prompt-and-mask loop with the reference implementation.

import numpy as np
from segment_anything import sam_model_registry, SamPredictor

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h.pth")
predictor = SamPredictor(sam)
predictor.set_image(image_rgb)               # heavy encoder runs ONCE here

# Now prompt as many times as you like; each call uses only the fast decoder.
point = np.array([[420, 310]])               # one (x, y) click on the target object
label = np.array([1])                        # 1 = foreground point, 0 = background
masks, scores, _ = predictor.predict(
    point_coords=point, point_labels=label,
    multimask_output=True,                   # return 3 candidate masks at different scales
)
best = masks[scores.argmax()]                # pick the highest-confidence mask
print("masks returned:", masks.shape[0], "| best mask covers", int(best.sum()), "pixels")
# masks returned: 3 | best mask covers 18742 pixels

3. Grounded Pipelines: Detect, Then Segment Advanced

SAM is powerful but, as the section noted, it does not know category names; it segments what you point at. An open-vocabulary detector knows names but outputs boxes, not pixel-precise masks. The natural and now ubiquitous combination is to chain them: an open-vocabulary detector (commonly Grounding DINO) takes a text query, finds the boxes of every matching object, and feeds those boxes as prompts to SAM, which returns clean masks. The result, often called Grounded SAM, lets you type a phrase and get pixel-accurate masks of everything matching it, with no task-specific training. This composition is the practical face of the whole chapter: foundation models as interoperable primitives.

# Conceptual pipeline: text query -> open-vocab boxes -> SAM masks.
# (Using grounded-sam style components; APIs vary by package version.)
text_query = "every dog . a red frisbee ."        # period-separated phrases, open vocabulary

# 1) Open-vocabulary detection: phrase -> boxes (region-text alignment of Section 1).
boxes, phrases, confidences = grounding_dino.predict(image_rgb, text_query)

# 2) Promptable segmentation: each detected box becomes a SAM box-prompt.
predictor.set_image(image_rgb)                      # encode image once
masks = []
for box in boxes:
    m, scores, _ = predictor.predict(box=box, multimask_output=False)
    masks.append(m[0])

print(f"query matched {len(boxes)} objects: {phrases}")
# query matched 3 objects: ['a dog', 'a dog', 'a red frisbee']

The practical example below shows a team replacing a brittle, labeled-from-scratch segmentation project with exactly this composition.

# Run open-vocabulary detection through a single Hugging Face pipeline:
# a text label list in, matching boxes out, no training and no plumbing.
from transformers import pipeline
# Open-vocabulary detection then mask generation, two pretrained models, no training:
detector = pipeline(model="IDEA-Research/grounding-dino-tiny", task="zero-shot-object-detection")
detections = detector(image, candidate_labels=["a dog", "a red frisbee"])
# Each detection's box can be passed to a SAM pipeline for a pixel-accurate mask.