Two-Stage Detectors: The R-CNN Family

"My method is simple. First I draw two thousand rectangles where something might be. Then I look at each one very carefully and decide what it is. It is slow, it is thorough, and it has never once been accused of missing the obvious."

A Region Proposal Network With a Strong Work Ethic

In Section 23.1 we established that a detector must emit a variable-length set of (box, class, confidence) triples and that mAP grades that set. The R-CNN family was the first deep approach to produce such a set, and it did so with an idea borrowed from common sense: do not try to classify every possible box in an image (there are millions), instead first find a few thousand regions that look like they might hold an object, then spend your expensive classifier only on those. That two-stage decomposition, propose then classify, is the subject of this section, and it remains the accuracy benchmark that the faster families of later sections are measured against. The illustration below pictures the two stages as a tidy assembly line.

1. R-CNN: Region Proposals Meet Deep Features Beginner

The original R-CNN (Girshick et al., 2014) combined three off-the-shelf pieces. First, a class-agnostic region proposal algorithm called Selective Search generated roughly 2,000 candidate boxes per image by merging superpixels of similar color and texture, a purely classical grouping method in the spirit of the classical segmentation of Chapter 11. Second, each proposed region was cropped, warped to the CNN's fixed input size ($227 \times 227$ for the AlexNet-style network the original used), and pushed through a CNN (an ImageNet-pretrained classifier, exactly the transfer learning idea of Chapter 21) to extract a feature vector. Third, a per-class linear support vector machine scored each feature vector, and a separate linear regressor refined the box coordinates.

R-CNN nearly doubled the previous best detection mAP on PASCAL VOC, which is why it mattered. But it was painfully slow: 2,000 forward passes of a deep CNN per image, with no computation shared between overlapping proposals, took tens of seconds per image at test time and required caching gigabytes of features to disk during training. The next two papers are the story of removing that waste, and the lesson generalizes far beyond detection: when an expensive computation is repeated on overlapping inputs, share it.

2. Fast R-CNN: Share the Backbone, Pool the Features Intermediate

Fast R-CNN's insight was that the 2,000 proposals all live in the same image, so the backbone should run once on the whole image to produce a single convolutional feature map, and each proposal should simply read out the portion of that map covering its region. Why does the feature have to be a fixed size at all? Because the second-stage head ends in fully-connected layers, and a fully-connected layer has a fixed weight matrix that only accepts a fixed-length input; proposals, however, come in every size and aspect ratio. The operation that bridges that mismatch, reading a fixed-size feature from a variable-size region, is RoI pooling (region-of-interest pooling): project the proposal box onto the feature map, divide it into a fixed grid (say $7 \times 7$), and max-pool within each grid cell, producing a $7 \times 7$ feature regardless of the proposal's size. That fixed-size feature then goes to a small head with two outputs: a softmax over classes (including a background class) and a per-class box refinement.

The arithmetic of that one change is the aha of the whole family. R-CNN pushed all 2,000 overlapping crops through the deep backbone separately, so two boxes sharing 90 percent of their pixels still paid for the backbone twice; Fast R-CNN runs the backbone exactly once on the full image and lets all 2,000 proposals read out of that single feature map, turning roughly 2,000 backbone passes into one. That is the difference between tens of seconds and a fraction of a second per image, from the same network, with the only new idea being "do not recompute what you can share." Sharing the backbone also let the whole second stage train end-to-end with a single multi-task loss. That loss is the template every detector since has used: a classification term plus a localization term, the second applied only to non-background boxes.

Here $p$ is the predicted class distribution, $u$ is the true class ($0$ for background), $t^u$ is the predicted box refinement for the true class, $v$ is the target refinement, $[u \ge 1]$ is $1$ for any object class and $0$ for background (so background boxes contribute no localization loss), and $\lambda$ balances the two terms. $L_{\text{loc}}$ is a smooth-L1 (Huber) loss on the four box-offset coordinates, which is robust to the outliers that a plain L2 would over-penalize.

3. Anchors: Turning a Grid into Box Candidates Intermediate

Faster R-CNN's contribution was to replace the slow external Selective Search with a learnable proposal network, and to make that work the field introduced the anchor box, an idea so central that the next two sections are partly defined by their relationship to it. An anchor is a reference box of a fixed size and aspect ratio, placed at every spatial location of the feature map. At each location you place several anchors (typically three scales times three aspect ratios, so nine anchors per location), tiling the image with a dense, fixed set of candidate boxes. The network does not predict boxes from nothing; it predicts, for each anchor, two things: an objectness score (does this anchor contain an object?) and four offsets that nudge the anchor toward a nearby ground-truth box.

The offsets are parameterized relative to the anchor, which keeps the regression targets small and scale-invariant. For an anchor with center $(x_a, y_a)$, width $w_a$, and height $h_a$, the predicted box $(x, y, w, h)$ comes from network outputs $(t_x, t_y, t_w, t_h)$ via

Multiplying $t_x$ by the anchor width $w_a$ is what makes the targets scale-invariant: a raw nudge of "ten pixels right" means a lot for a tiny anchor and almost nothing for a huge one, but a nudge of "0.1 anchor-widths right" means the same proportional shift at every size, so the network can reuse one set of weights across scales. The exponential on width and height guarantees a positive size and lets the network express large size changes with small outputs. Anchors are matched to ground truth by IoU (the metric from Section 23.1): an anchor with IoU above $0.7$ to some ground-truth box is a positive example, below $0.3$ is a negative (background) example, and the rest are ignored during training. Figure 23.2.1 shows nine anchors tiled at one feature-map location.

4. Faster R-CNN: The Region Proposal Network Intermediate

The region proposal network (RPN) is a tiny convolutional head that slides over the shared backbone feature map and, at every location, outputs the objectness scores and box offsets for the $k$ anchors there. It is a fully convolutional, class-agnostic object detector in its own right; its only job is to say "object here, roughly this box" or "background". Its top-scoring proposals (after a non-maximum suppression pass, described below) become the regions that the Fast R-CNN second stage then classifies into actual categories and refines. Because the RPN shares the backbone with the second stage, proposals are nearly free, and the whole detector is one network trained with a combined RPN-plus-detection loss. Figure 23.2.2 traces the data flow.

Both stages produce many overlapping boxes per object, so both apply non-maximum suppression (NMS): sort boxes by score, keep the top box, discard every remaining box whose IoU with it exceeds a threshold (commonly $0.5$ to $0.7$), and repeat. NMS is the duplicate-killer that the evaluation rule of Section 23.1 demanded, and it is the one piece of every detector in this section and the next that is not learned. The from-scratch implementation below is worth reading, because removing this hand-tuned step is precisely what DETR achieves in Section 23.5.

# Greedy non-maximum suppression: repeatedly keep the highest-scoring box and
# discard every remaining box that overlaps it more than iou_thresh.
# This is the one hand-tuned, non-learned step in a classic detector.
import torch

def nms(boxes, scores, iou_thresh=0.5):
    """boxes: (N, 4) corner format. scores: (N,). Returns kept indices, by score."""
    keep = []
    idxs = scores.argsort(descending=True)        # highest score first
    while idxs.numel() > 0:
        i = idxs[0].item()
        keep.append(i)
        if idxs.numel() == 1:
            break
        # IoU of the kept box against all remaining candidates.
        x1 = boxes[idxs[1:], 0].clamp(min=boxes[i, 0])
        y1 = boxes[idxs[1:], 1].clamp(min=boxes[i, 1])
        x2 = boxes[idxs[1:], 2].clamp(max=boxes[i, 2])
        y2 = boxes[idxs[1:], 3].clamp(max=boxes[i, 3])
        inter = (x2 - x1).clamp(min=0) * (y2 - y1).clamp(min=0)
        area_i = (boxes[i, 2] - boxes[i, 0]) * (boxes[i, 3] - boxes[i, 1])
        area_r = ((boxes[idxs[1:], 2] - boxes[idxs[1:], 0]) *
                  (boxes[idxs[1:], 3] - boxes[idxs[1:], 1]))
        iou = inter / (area_i + area_r - inter)
        idxs = idxs[1:][iou <= iou_thresh]         # drop boxes that overlap the kept one
    return torch.tensor(keep)

boxes = torch.tensor([[10, 10, 50, 50], [12, 11, 51, 49], [90, 90, 130, 130]],
                     dtype=torch.float)
scores = torch.tensor([0.9, 0.8, 0.7])
print(nms(boxes, scores, 0.5).tolist())   # [0, 2]  (box 1 suppressed as a duplicate of 0)

# Load a COCO-pretrained Faster R-CNN from torchvision and run inference.
# The single constructor pulls in the backbone, FPN, RPN, RoI Align, the
# two-stage head, and a fused batched NMS; we just threshold the scores.
import torch
from torchvision.models.detection import (
    fasterrcnn_resnet50_fpn_v2, FasterRCNN_ResNet50_FPN_V2_Weights)

weights = FasterRCNN_ResNet50_FPN_V2_Weights.DEFAULT
model = fasterrcnn_resnet50_fpn_v2(weights=weights).eval()
preprocess = weights.transforms()                # resize + normalize, COCO-style

img = torch.rand(3, 800, 800)                    # a CHW float image in [0, 1]
with torch.no_grad():
    out = model([preprocess(img)])[0]            # dict: boxes, labels, scores
keep = out["scores"] > 0.5
print(out["boxes"][keep].shape, out["labels"][keep].tolist())

5. Why It Is Accurate but Slow Advanced

The two-stage design is accurate for a structural reason: the second stage sees only a few hundred high-quality proposals, each already roughly localized, and can spend a heavyweight per-region head deciding the exact class and tightening the box. The proposals also rebalance the data; the RPN winnows millions of anchors down to a few thousand, and a sampling step keeps the ratio of foreground to background examples in the second stage near $1{:}3$ rather than the $1{:}1000$ that a one-stage detector must confront. That balance is why two-stage detectors did not need the focal loss that Section 23.3 invents to rescue one-stage training.

The cost is the same structure. The per-region head runs hundreds of times per image, the proposal and NMS steps are sequential and hard to fully parallelize, and the whole pipeline has more moving parts (anchor matching, proposal sampling, two NMS passes) to tune. A Faster R-CNN runs at perhaps 5 to 15 frames per second on a strong GPU, fine for asynchronous analysis but too slow for a phone, a drone, or a self-driving car's perception loop. That speed ceiling is exactly the gap the one-stage detectors of the next section were built to close, and the focal-loss innovation that let them close it without surrendering accuracy is the story we turn to now.