One-Stage Detectors: YOLO, SSD & RetinaNet

"Why propose regions and then judge them, like some indecisive committee? I look once. The whole grid speaks at the same instant, every cell shouting what it sees and where. It is chaos, but it is fast chaos, and I have learned to ignore the ten thousand cells yelling 'background' at me."

A YOLO Grid Cell That Made Up Its Mind

The Faster R-CNN of Section 23.2 is accurate but spends a per-region head on hundreds of proposals, capping it at a handful of frames per second. For a self-driving perception stack or a phone camera that wants to draw boxes on a live preview, that is far too slow. The one-stage family asks: what if we skip the proposal stage and predict everything in a single forward pass, treating detection as a dense regression problem over a fixed grid? The answer is a class of detectors an order of magnitude faster, and the engineering question that dominates them is how to train such a dense predictor when almost every location is background.

1. YOLO: You Only Look Once Beginner

The original YOLO (Redmon et al., 2016) reframed detection as a single regression. Divide the image into an $S \times S$ grid; each cell predicts a small fixed number of boxes (center, size, and an objectness confidence) plus one class distribution for the cell. A whole image's worth of detections, all boxes and all classes, emerges from one forward pass of one network, hence "you only look once." The training target assigns each ground-truth object to the single grid cell containing its center, and the loss is a weighted sum of localization error on the responsible box, objectness error everywhere, and classification error on the responsible cell.

That single-pass design made YOLO dramatically faster than anything before it, running at real-time frame rates while staying respectably accurate. Its weaknesses were equally instructive: the original coarse grid struggled with small objects and with multiple objects whose centers fell in the same cell, and its single-scale prediction missed the multi-scale structure that detection needs. Every YOLO version since (and there have been many, through YOLO11 in 2024 and the YOLO12 line in 2025) has been a steady accumulation of the fixes the rest of this chapter describes: anchors, then anchor-free heads, multi-scale feature pyramids, better losses, and stronger augmentation. The YOLO name now denotes a family of fast, well-engineered one-stage detectors rather than a single architecture, and it is the family you will fine-tune in Section 23.6.

2. SSD: Multi-Scale Default Boxes Intermediate

SSD (Single Shot MultiBox Detector, Liu et al., 2016) addressed YOLO's single-scale weakness head-on. Instead of predicting from one final feature map, SSD attaches detection heads to several feature maps of decreasing resolution within the backbone. Early, high-resolution maps have small receptive fields and detect small objects; later, low-resolution maps have large receptive fields and detect large objects. At each location of each map, SSD places a set of default boxes (its name for anchors) of a few aspect ratios, and predicts a class score and a box offset per default box, exactly the anchor parameterization of Section 23.2. The result is a dense, multi-scale, single-pass detector that handled the range of object sizes far better than the original YOLO.

SSD's multi-scale insight, that different feature-map resolutions should detect different object sizes, is the direct ancestor of the feature pyramid network (FPN) that nearly every modern detector uses. FPN improves on SSD's naive use of backbone maps by adding a top-down pathway: it takes the semantically rich but spatially coarse deep feature map, upsamples it, and adds it to the spatially fine but semantically weak shallow maps, so that every level of the pyramid is both high-resolution and semantically strong. This is the learned, end-to-end descendant of the Gaussian and Laplacian image pyramids you built in Chapter 4, and the feature-hierarchy fusion of Chapter 20. Figure 23.3.1 contrasts the single-scale, plain multi-scale, and top-down-fused designs.

3. The Foreground-Background Imbalance Intermediate

Here is the structural problem that kept one-stage detectors behind. A dense detector evaluates a class score at every anchor on every pyramid level, perhaps a hundred thousand anchors per image. Of those, only a few dozen overlap a real object; the rest, more than 99.9 percent, are background. When you sum a standard cross-entropy loss over all of them, the gradient is dominated by the enormous mass of easy background examples, each contributing a tiny but relentless signal that collectively drowns out the few foreground examples that actually teach the model what objects look like. Two-stage detectors dodged this because the RPN and a sampling step rebalanced the data before the second-stage loss; a one-stage detector has no such filter. The illustration below previews the fix: turning the easy crowd's volume down rather than discarding it.

The standard cross-entropy for a single example with predicted probability $p_t$ of the correct class is $\text{CE}(p_t) = -\log(p_t)$. The trouble is that even a well-classified background example with, say, $p_t = 0.9$ still contributes a loss of $-\log(0.9) \approx 0.105$, and multiplied by a hundred thousand such examples this is a large, persistent gradient that swamps the foreground. The detector learns to predict "background" confidently and stops improving on the rare objects.

4. Focal Loss: Down-Weighting the Easy Advanced

The focal loss (Lin et al., 2017) is cross-entropy multiplied by a modulating factor that shrinks toward zero as an example becomes easy:

The factor $(1 - p_t)^{\gamma}$ is the whole idea. For a well-classified example $p_t$ is near $1$, so $(1 - p_t)^{\gamma}$ is near $0$ and the example's loss is suppressed; for a misclassified example $p_t$ is small, $(1 - p_t)^{\gamma}$ is near $1$, and the loss is essentially unchanged. The focusing parameter $\gamma$ (the paper uses $\gamma = 2$) controls how aggressively easy examples are down-weighted: at $\gamma = 0$ focal loss is plain cross-entropy, and larger $\gamma$ silences the easy examples harder. The $\alpha_t$ is an ordinary class-weighting term that additionally balances foreground against background. Figure 23.3.2 plots how the modulating factor collapses the loss of easy examples.

The implementation is a few lines on top of the binary-cross-entropy logits loss, and reading it makes the modulating factor concrete.

# Focal loss: down-weight the easy, well-classified examples so the hundred
# thousand background anchors stop drowning out the rare foreground ones.
# The (1 - p_t) ** gamma factor is what does the rebalancing.
import torch
import torch.nn.functional as F

def focal_loss(logits, targets, alpha=0.25, gamma=2.0):
    """logits, targets: same shape (e.g. (N, num_classes)) with 0/1 targets.
       Returns the summed focal loss over all entries."""
    p = torch.sigmoid(logits)
    ce = F.binary_cross_entropy_with_logits(logits, targets, reduction="none")
    p_t = p * targets + (1 - p) * (1 - targets)        # prob. of the true label
    modulating = (1 - p_t) ** gamma                    # collapses for easy examples
    alpha_t = alpha * targets + (1 - alpha) * (1 - targets)
    return (alpha_t * modulating * ce).sum()

logits = torch.tensor([[3.0], [0.1]])     # one easy positive, one hard positive
targets = torch.tensor([[1.0], [1.0]])
print(focal_loss(logits, targets).item())  # the easy example contributes far less

RetinaNet is the architecture that paired this loss with a ResNet-plus-FPN backbone and a simple dense head. With the imbalance solved by the loss rather than by sampling, RetinaNet became the first one-stage detector to match the accuracy of the best two-stage models, while keeping the one-stage speed advantage. Focal loss has since spread far beyond RetinaNet; it appears in FCOS (Section 23.4), in many YOLO heads, and in the classification branch of several DETR variants.

# Load a COCO-pretrained RetinaNet and run a forward pass.
# The same focal loss from Code Fragment 1 is available fused as
# torchvision.ops.sigmoid_focal_loss, so nothing here is hand-written.
import torch
from torchvision.models.detection import (
    retinanet_resnet50_fpn_v2, RetinaNet_ResNet50_FPN_V2_Weights)
from torchvision.ops import sigmoid_focal_loss     # the same loss, fused

weights = RetinaNet_ResNet50_FPN_V2_Weights.DEFAULT
model = retinanet_resnet50_fpn_v2(weights=weights).eval()
with torch.no_grad():
    out = model([torch.rand(3, 800, 800)])[0]       # boxes, labels, scores
print({k: v.shape for k, v in out.items()})

5. Choosing Among the One-Stage Family Advanced

The one-stage detectors share a profile: a single forward pass, dense prediction on FPN levels, a focal or task-aligned classification loss to handle imbalance, and an NMS post-step (which the newest YOLOs are learning to drop). They differ mainly in the backbone size, the head design, and the training recipe, and the practical choice is a speed-accuracy trade-off rather than a deep architectural decision. A modern compact YOLO is the default for edge and real-time work; RetinaNet remains a clean, well-understood research baseline; SSD is largely of historical interest now, superseded by its descendants. Table 23.3.1 summarizes the trade-offs against the two-stage family of the previous section.

Both families still rely on the anchor box, the hand-designed reference shapes whose scales and aspect ratios you must tune to your data, as Exercise 23.2.2 showed. That tuning is a real burden, and it is the next thing the field shed. Section 23.4 shows how detectors learned to predict boxes directly from feature-map locations with no anchors at all.