Anchor-Free & Keypoint-Based Detection

"For years they made me memorize a catalogue of rectangles before I was allowed to look at anything. Nine shapes per spot, all of them guesses about objects I had not yet seen. Then one day they said: just point at the thing and tell me how far the edges are. I have never felt so unburdened."

An Anchor Box, Finally Retired

The anchor box served the detectors of Section 23.2 and Section 23.3 well, but Exercise 23.2.2 exposed its hidden cost: you must choose anchor scales and aspect ratios that cover your objects, and a poor choice silently caps the best achievable IoU. Worse, anchors introduce an intricate label-assignment step (which anchor is responsible for which object?) full of IoU thresholds and sampling ratios. Around 2019 the field asked whether anchors were necessary at all, given that a feature map already tiles the image with locations. The answer, delivered by FCOS and CenterNet, was no: a location can predict an object directly. This section is about how, and about why the resulting simplicity won. The illustration below captures the shift, from a heavy catalogue of reference boxes to simply pointing at the edges.

1. FCOS: Per-Location Box Regression Intermediate

FCOS (Fully Convolutional One-Stage detector, Tian et al., 2019) makes the simplest possible anchor-free choice. For every location $(x, y)$ on an FPN feature map, it asks two questions. First, classification: does this location fall inside some ground-truth object's box, and if so, of what class? Second, regression: if it does, how far is this location from the four edges of that box? The box is encoded as four positive distances $(l, t, r, b)$, the distances from the location to the left, top, right, and bottom edges. Decoding is trivial: for a location at $(x, y)$ the predicted box corners are $(x - l, y - t, x + r, y + b)$. There are no anchors, no aspect ratios, and no IoU-based anchor matching; a location is a positive training example simply if it lands inside a ground-truth box.

Two complications need handling. A location near the boundary of two overlapping objects could fall inside both; FCOS resolves this with the FPN, assigning each object size to a specific pyramid level so that conflicting boxes rarely land on the same level. The decode logic is so simple it fits in a few lines, shown below for one feature-map level.

# FCOS decode: turn a feature-map location plus its four predicted edge
# distances (left, top, right, bottom) into a corner-format box.
# No reference anchors and no exponential, just subtract and add the distances.
import torch

def fcos_decode(locations, ltrb):
    """locations: (N, 2) pixel coords of feature-map points.
       ltrb: (N, 4) predicted distances to left, top, right, bottom edges.
       Returns (N, 4) corner-format boxes."""
    x, y = locations[:, 0], locations[:, 1]
    l, t, r, b = ltrb[:, 0], ltrb[:, 1], ltrb[:, 2], ltrb[:, 3]
    x1 = x - l
    y1 = y - t
    x2 = x + r
    y2 = y + b
    return torch.stack([x1, y1, x2, y2], dim=1)

locs = torch.tensor([[250.0, 150.0]])
ltrb = torch.tensor([[126.0, 86.0, 146.0, 76.0]])   # distances from the figure
print(fcos_decode(locs, ltrb))   # [[124., 64., 396., 226.]]  recovers the GT box

2. The Center-ness Trick Advanced

A subtle problem appears: locations near the edge of an object regress long, lopsided distances and produce sloppy boxes, yet during training they are treated as fully positive examples, so the model emits many low-quality boxes from off-center locations. FCOS fixes this with a third output, a center-ness score that the network predicts at each location and that measures how centered the location is within its object. Center-ness is defined from the regression targets as

This is $1$ at the exact center of the box (where $l = r$ and $t = b$) and decays toward $0$ near the edges. At inference, the final box score is the classification score multiplied by the center-ness. A confidently-classified box from a corner location therefore has high class score but low center-ness, so its combined score is low. That pushes the off-center boxes down the ranking, and NMS keeps the well-centered, well-localized ones. Center-ness is a small branch with an outsized effect; it is the anchor-free analogue of the objectness score, and it is why FCOS matched anchor-based RetinaNet despite its radical simplification.

3. CenterNet: Objects as Heatmap Peaks Advanced

CenterNet (Zhou et al., 2019, "Objects as Points") takes anchor-free thinking to its logical conclusion: represent each object by a single point, its center, and detect objects as peaks in a class-specific heatmap. The network outputs, for each class, a low-resolution heatmap whose value at a pixel is high where that class's object center is. Training renders each ground-truth center as a Gaussian blob on the heatmap (the same Gaussian kernel you met in the spatial filtering of Chapter 3) and regresses the network output toward it with a focal-style loss. Separate regression maps predict, at each center pixel, the object's width and height and a small offset that recovers the precision lost to the heatmap's downsampling. At inference, you simply take the local maxima of each heatmap (a $3 \times 3$ max-pool finds peaks), read off the size and offset at each peak, and you have your boxes, with no NMS at all, because a peak-picking heatmap inherently produces one detection per object.

This framing is elegant and influential beyond box detection: the same heatmap-of-keypoints idea is how human pose estimation localizes joints (each joint is a keypoint peak), and CenterNet itself extends naturally to predicting any set of points per object. It connects detection to the keypoint and descriptor ideas of Chapter 10, where you found interest points as extrema of a response map; CenterNet learns the response map end-to-end and makes its peaks the object centers. Figure 23.4.2 contrasts the three detection paradigms of this chapter at the level of what each predicts per location.

# Load a COCO-pretrained anchor-free FCOS detector and run inference.
# The center-ness branch, per-location regression, and FPN-level size
# assignment all live inside the single constructor.
import torch
from torchvision.models.detection import fcos_resnet50_fpn, FCOS_ResNet50_FPN_Weights

weights = FCOS_ResNet50_FPN_Weights.DEFAULT
model = fcos_resnet50_fpn(weights=weights).eval()
with torch.no_grad():
    out = model([torch.rand(3, 800, 800)])[0]    # boxes, labels, scores
keep = out["scores"] > 0.5
print(out["boxes"][keep].shape)

4. Why the Simplification Mattered Advanced

Anchor-free detection removed three sources of friction at once. It deleted the anchor hyperparameters (scales, ratios, counts) that had to be re-tuned per dataset. It deleted the anchor-to-ground-truth IoU matching step and its thresholds. And it reduced the number of predictions per location, which lightens the head and speeds inference. The accuracy turned out to be at least as good once center-ness or a good label-assignment rule replaced the role anchors had played in deciding which predictions to trust. The net effect is a cleaner, more portable detector that adapts to new object-shape distributions without engineering, which is why the family became the foundation of the modern real-time YOLOs.

But notice what every detector in this chapter so far still shares: they all emit many overlapping candidate boxes and rely on a hand-designed NMS to clean up duplicates (CenterNet's peak-picking is the partial exception). NMS is a non-differentiable, separately-tuned post-process that sits outside the network and can merge or split genuine objects in crowded scenes. The next section presents DETR, which removes NMS entirely by training the network to predict a clean set in the first place, using the transformer you built in Chapter 22 and a matching loss that makes duplicate suppression part of learning rather than a postscript.