Detection & Segmentation Frameworks: Detectron2 & MMDetection

"They told me I could be any detector I wanted. Then they handed me a config file with four hundred lines and said the architecture was on line two hundred and twelve, inherited from a base I would have to go and read."

A Two-Stage Detector Assembled Entirely From YAML

Big Picture

When an Ultralytics one-liner cannot express the detector you need, you move to a framework that treats a model as a composition of swappable parts described by a config file, and the two that dominate research are Detectron2 and MMDetection. Both decompose a detector into backbone, neck, and head, register every component by name, and let a text config select and wire them. The price of that flexibility is a learning curve: you trade three lines of YOLO for a config system you must learn to read. The payoff is reproducibility and control: a published result is a config you can run, and a new architecture is a few lines of config away. The illustration below shows the snap-together view of a detector that both frameworks adopt.

The moment you need a detector Ultralytics does not ship, a Swin backbone under a Cascade R-CNN head, a custom anchor scheme, a published architecture from last month's paper, the one-line API runs out of room and you are stuck. Section 29.1 ended exactly at that boundary of convenience. This section is about what you reach for instead. Detectron2 (from Meta) and MMDetection (from OpenMMLab) are the two heavyweight frameworks the research community uses to build, train, and benchmark detectors and segmenters. They share a design philosophy, decompose a model into registered, swappable modules driven by a config, and differ mostly in style and breadth. We read both config systems, compare their zoos, and end with a guide to choosing between them and the lighter tools of the previous section.

1. Why a Framework, Not a Script Intermediate

Recall the anatomy of a detector from Chapter 23: a backbone extracts features, a neck (typically a Feature Pyramid Network, or FPN) fuses them across scales, and one or more heads predict boxes and classes. A segmenter from Chapter 24 adds a mask head. The insight both frameworks exploit is that these parts are interchangeable. You can pair any backbone with any neck and any head, and most detection research is precisely such recombination. A framework that makes the parts named, registered, and config-selectable turns "try Swin instead of ResNet" from a code rewrite into a one-line config edit. Figure 29.2.1 shows the decomposition that both frameworks encode.

2. Detectron2: The Registry and the Config

Detectron2 is Meta's rewrite of the original Detectron, built on PyTorch. Its core abstraction is the registry: a lookup table that maps a string name to the Python class that builds a component, so that naming "ResNet" in a text file is enough for the framework to find and construct the right module without any import or code edit. Think of the registry as a restaurant menu: you write "ResNet" on the order slip the way a diner writes a dish name, and the kitchen (the framework) already knows which recipe (Python class) to cook, so you never touch the recipe yourself and swapping your order to "Swin" is just writing a different name, not rewriting the kitchen. The analogy stops at the door of a brand-new dish: a component nobody has registered is a dish not on the menu, which is why adding one still means writing the class and decorating it so the registry learns its name. Every backbone, head, and loss is registered under such a name, and a CfgNode config (a nested key-value tree) selects them. The library ships a model zoo of Faster R-CNN, Mask R-CNN, RetinaNet, and panoptic models, each defined by a config you can load, run, or edit. The fastest entry is the zoo plus a DefaultPredictor, which loads a config and weights and runs inference.

# Run Detectron2 inference straight from a model-zoo config: load a complete
# Mask R-CNN definition, edit one threshold on the config tree, then let a
# DefaultPredictor instantiate the model and return instance masks and classes.
from detectron2 import model_zoo
from detectron2.config import get_cfg
from detectron2.engine import DefaultPredictor
import cv2

# Start from a zoo config: Mask R-CNN with an R-50 FPN backbone, COCO-trained.
cfg = get_cfg()
zoo = "COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"
cfg.merge_from_file(model_zoo.get_config_file(zoo))
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(zoo)   # download trained weights
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5             # confidence threshold

predictor = DefaultPredictor(cfg)
image = cv2.imread("street.jpg")                        # BGR, as OpenCV loads it
outputs = predictor(image)
instances = outputs["instances"]
print(instances.pred_classes)   # tensor of class indices
print(instances.pred_masks.shape)   # [N, H, W] boolean instance masks

Customization happens by editing the config tree. To swap the backbone, you change cfg.MODEL.BACKBONE.NAME and the matching cfg.MODEL.RESNETS block; to train on your own data, you register a dataset (in COCO format, the standard from Chapter 23) and point cfg.DATASETS.TRAIN at it. The registry pattern means adding a brand-new head is also possible: you write the module, decorate it with @ROI_HEADS_REGISTRY.register(), and name it in the config. This is the control that the Ultralytics one-liner cannot give you.

3. MMDetection: Inheritance-Based Configs Advanced

MMDetection, part of the OpenMMLab family, shares Detectron2's registry-and-config philosophy but pushes the config system further: configs inherit from base configs through a _base_ list, so a new model is often a short file that imports a base and overrides a few keys. Its model zoo is the broadest published anywhere, hundreds of detectors and segmenters spanning two-stage, single-stage, anchor-free, and transformer-based (DETR, Deformable DETR, DINO) designs, which makes it the reference for comparing architectures on equal footing.

# A custom MMDetection config: inherit a full Mask R-CNN, override a few fields.
# Saved as configs/my_mask_rcnn.py
_base_ = [
    "../_base_/models/mask-rcnn_r50_fpn.py",       # the model architecture
    "../_base_/datasets/coco_instance.py",          # dataset + pipeline
    "../_base_/schedules/schedule_1x.py",           # optimizer + LR schedule
    "../_base_/default_runtime.py",                 # logging, checkpoints
]

# Override: swap the ResNet-50 backbone for a Swin-Tiny transformer backbone.
model = dict(
    backbone=dict(
        _delete_=True,                              # drop the inherited backbone
        type="SwinTransformer",
        embed_dims=96, depths=[2, 2, 6, 2], num_heads=[3, 6, 12, 24],
    ),
    neck=dict(in_channels=[96, 192, 384, 768]),     # match Swin's channel widths
)

# Reduce the number of classes for a 3-class custom dataset.
model.update(roi_head=dict(
    bbox_head=dict(num_classes=3), mask_head=dict(num_classes=3)))

The inheritance model is powerful and, at first, confusing: to understand a config you must trace its _base_ chain, and the architecture you are editing may be defined two files away. The payoff is that swapping a CNN backbone for the Swin transformer of Chapter 22, or swapping a two-stage head for a DETR-style set-prediction head, really is a short override. MMDetection's breadth makes it the natural home when you want to benchmark a new idea against many baselines under one training harness.

Table 29.2.1 captures the practical split. Detectron2's flatter configs and curated zoo make it the gentler framework for someone who wants a strong Mask R-CNN with a clean Python API; MMDetection's inheritance and enormous zoo make it the reference for research that compares many architectures or needs a transformer-based detector that Detectron2 does not ship.

4. The mAP Connection

Both frameworks report results in mean Average Precision, the detection metric defined in Chapter 23, and they compute it identically because both adopt the official COCO evaluation. Recall that mAP averages precision over recall levels and over Intersection-over-Union thresholds; COCO's primary metric averages over IoU from $0.5$ to $0.95$ in steps of $0.05$, written $\text{mAP}_{0.5:0.95}$.

The strictness of that range is easy to underestimate, so trace one box through it. Suppose a predicted box overlaps the true box at IoU $0.6$. It counts as a correct detection at the $0.5$ threshold, but it misses at every threshold from $0.7$ upward, so it earns credit in only 3 of the 10 averaged thresholds. That visually-decent box therefore scores barely above 0.3 on this metric. That is why a number like $40$ percent $\text{mAP}_{0.5:0.95}$ is a strong result, not a failing grade: most of the "missing" 60 points are boxes that look right but are not pixel-tight.

When you compare a config you trained against a published number, you are comparing on this exact protocol, which is why framework-reported numbers are directly comparable across papers in a way that ad-hoc evaluation scripts never are. This standardization is itself a reason to use a framework rather than roll your own evaluation.

5. Choosing Among the Three Tiers

You now have three tiers for detection and segmentation: the Ultralytics one-liner from Section 29.1, and the two frameworks here. The decision ladder is short. If a YOLO architecture meets your accuracy and latency needs and you want results today, use Ultralytics. If you need a non-YOLO architecture, a custom component, or a faithful reproduction of a published config, use a framework. Between the frameworks, choose Detectron2 for clean R-CNN-family work and MMDetection for broad benchmarking or transformer detectors. The cost rises with the tier, more concepts, more config to read, but so does control, and the right tier is the lowest one that can express what you need.

6. Summary

When convenience runs out, you reach for a framework that treats a detector as a composition of registered, config-selectable parts. Detectron2 offers a flat config and a curated R-CNN-family zoo with a clean API; MMDetection offers inheritance-based configs and the broadest zoo, including transformer detectors. Both decompose models into backbone, neck, and head, both evaluate on the official COCO mAP that makes results comparable, and both turn a custom detector from a thousand-line build into a twenty-line config edit. Choose the lowest tier that expresses your need. With models loaded and detectors composed, the remaining bottleneck is rarely the model; it is the data. Section 29.3 turns to the annotation, versioning, and visual-debugging tooling that decides your real accuracy ceiling.