Panoptic Segmentation: Unifying Things & Stuff

"Semantic segmentation labeled the sky and forgot the birds. Instance segmentation counted the birds and ignored the sky. Panoptic segmentation, finally, labeled every last pixel and could still tell me there were exactly three birds. It is the only one I trust to describe a whole scene."

A Scene Parser With No Pixels Left Behind

Ask either tool you have built so far to describe a street and it answers half the question. Section 24.1 gives a class to every pixel but no sense of separate objects; Section 24.2 gives a separate mask to every object but says nothing about the background and is happy to let masks overlap. Neither alone produces a complete scene description. A self-driving stack, a robot mapping a room, or a system captioning a photo wants both: where is the drivable road (stuff), and exactly which and how many pedestrians are on it (things), with every pixel accounted for. This complete-partition goal echoes the classical region-partitioning of Chapter 11, where watershed and graph cuts carved an image into non-overlapping regions by hand-designed energy rather than learned classes. Panoptic segmentation, defined by Kirillov and colleagues in 2019, is the task that demands this complete answer.

1. Things, Stuff, and the Panoptic Output Format Beginner

The vocabulary matters because it shapes the task. Things are categories whose instances you can count and point to: person, car, dog, bicycle. Stuff is categories with no natural instances, where counting makes no sense: sky, road, vegetation, water, wall. The same scene contains both, and the panoptic task treats them by one rule. Every pixel receives a pair $(c, k)$: a class label $c$, and an instance id $k$. For thing classes, $k$ distinguishes the individual objects (car 1, car 2). For stuff classes, $k$ is ignored, all pixels of "sky" share one region. The hard constraint is that the assignment is a partition: every pixel gets exactly one pair, so the segments are mutually exclusive and collectively exhaustive. Figure 24.3.1 shows how panoptic sits between the two tasks you know.

That partition constraint is exactly what neither earlier task enforces. Instance masks from Mask R-CNN can overlap, two boxes both claiming a pixel, and they leave the background unlabeled. Producing a valid panoptic map therefore requires a merge step that resolves conflicts and fills the gaps, which is the subject of subsection 2.

2. Merging Semantic and Instance Predictions Intermediate

The first panoptic systems were built by combining the two networks you already have: a semantic segmenter for stuff and an instance segmenter (Mask R-CNN) for things, with a heuristic merge that turns their outputs into a clean partition. The merge resolves three kinds of conflict. First, overlapping instance masks: sort instances by confidence and paint them in order, so a higher-confidence object wins any contested pixel. Second, thing-versus-stuff conflict: where an instance mask and a semantic stuff region disagree, the thing instance takes precedence (a person standing on the road is person, not road). Third, gaps: any pixel left unlabeled after instances are painted falls back to the semantic prediction, and very small stuff regions below a threshold are discarded as noise. The code below implements this canonical merge.

# The canonical heuristic merge that turns a semantic map plus instance masks
# into a valid panoptic partition: paint instances by descending confidence onto
# unclaimed pixels only, drop tiny fragments, then fill the gaps with stuff classes.
import torch

def merge_to_panoptic(sem_logits, inst_masks, inst_labels, inst_scores,
                      stuff_classes, score_thr=0.5, area_thr=64):
    """Combine a semantic map and instance masks into a non-overlapping panoptic map.
    Returns (class_map, instance_id_map): two HxW integer tensors.
    """
    H, W = sem_logits.shape[-2:]
    sem = sem_logits.argmax(0)                      # (H, W) semantic class per pixel
    class_map = torch.full((H, W), -1, dtype=torch.long)   # -1 = not yet assigned
    id_map    = torch.zeros((H, W), dtype=torch.long)
    next_id = 1

    # 1) Paint instances (things) in order of decreasing confidence.
    order = inst_scores.argsort(descending=True)
    for i in order:
        if inst_scores[i] < score_thr:
            continue
        m = (inst_masks[i] > 0.5) & (class_map == -1)  # only claim unassigned pixels
        if m.sum() < area_thr:
            continue                                   # drop tiny fragments
        class_map[m] = inst_labels[i]
        id_map[m] = next_id
        next_id += 1

    # 2) Fill remaining pixels with stuff from the semantic map.
    unassigned = class_map == -1
    for c in stuff_classes:
        sel = unassigned & (sem == c)
        class_map[sel] = c                             # stuff: shared id 0
    return class_map, id_map

# Toy run with dummy tensors.
sem_logits = torch.randn(20, 64, 64)                   # 20-class semantic logits
inst_masks  = torch.rand(3, 64, 64)                    # 3 instance soft masks
out_c, out_id = merge_to_panoptic(sem_logits, inst_masks,
                                  torch.tensor([1, 1, 5]),     # two people, one car
                                  torch.tensor([0.9, 0.8, 0.6]),
                                  stuff_classes=[10, 11, 12])
print(out_c.shape, "unique instances:", out_id.unique().numel() - 1)

This two-network-plus-merge approach works but is clumsy: two models to train, two sets of features, and a hand-tuned heuristic gluing them. The field's response was end-to-end panoptic networks. Panoptic FPN shares one backbone between a semantic head and an instance head; Panoptic-DeepLab uses a single bottom-up design with class and instance-center predictions. These reduced the redundancy, but the cleanest answer came from the mask transformers of Section 24.4, which predict the non-overlapping mask set directly and make the merge step disappear.

3. Panoptic Quality: One Number for Two Jobs Intermediate

A panoptic predictor can fail in two distinct ways: it can mislabel or miss a segment (a recognition error), or it can label the right segment with a sloppy boundary (a segmentation error). The panoptic quality (PQ) metric scores both in a single value. For each class, a predicted segment and a ground-truth segment are matched if their Intersection-over-Union (IoU, the same overlap measure from Chapter 23) exceeds 0.5. Because a strict-majority overlap can match at most one prediction to one truth, this matching is unique. PQ then combines the average IoU of the matched pairs with the precision and recall of the matching itself:

The decomposition is the metric's gift. SQ (segmentation quality) is the mean IoU over correctly matched segments: how tight are your boundaries when you do find a segment? RQ (recognition quality) is an F1-style score over the matches: how well do you find and not hallucinate segments? The half-weights on $|FP|$ and $|FN|$ are simply what makes this expression equal the F1 score, the harmonic mean of precision and recall, written compactly. Their product, PQ, is high only when you both find segments and delineate them well, and reading SQ and RQ separately tells you which of the two failure modes is hurting you. The metric is computed per class and averaged, and the literature usually also reports it split as $\text{PQ}^{\text{Th}}$ over thing classes and $\text{PQ}^{\text{St}}$ over stuff classes. Figure 24.3.2 makes the two factors concrete.

The IoU at the core of PQ is the same generalization of the metric you will formalize fully in Section 24.6, and it connects all the way back to the overlap and matching ideas you first met in the detection evaluation of Chapter 23. The minimal IoU helper below is the workhorse behind any PQ implementation.

# The mask-IoU helper at the heart of any panoptic-quality implementation.
# IoU is intersection over union of the set pixels in two boolean masks;
# the 0.5 threshold on this value decides whether a pair is a true positive.
import torch

def mask_iou(pred_mask, gt_mask):
    """IoU between two boolean HxW masks: intersection over union of set pixels."""
    inter = (pred_mask & gt_mask).sum().float()
    union = (pred_mask | gt_mask).sum().float()
    return (inter / union).item() if union > 0 else 0.0

a = torch.zeros(8, 8, dtype=torch.bool); a[2:6, 2:6] = True   # 4x4 square
b = torch.zeros(8, 8, dtype=torch.bool); b[3:7, 3:7] = True   # shifted 4x4 square
print(f"IoU = {mask_iou(a, b):.3f}")   # IoU = 0.391  (9 overlap / 23 union)

# Score panoptic quality with the official panopticapi evaluator instead of
# hand-rolling the IoU matching and per-class accumulation. It consumes
# id-encoded panoptic PNGs plus COCO-format JSON and returns PQ, SQ, RQ.
# (pip install git+https://github.com/cocodataset/panopticapi)
from panopticapi.evaluation import pq_compute

# Inputs are panoptic PNGs (id-encoded) plus matching JSON in COCO panoptic format.
results = pq_compute(gt_json_file="gt_panoptic.json",
                     pred_json_file="pred_panoptic.json",
                     gt_folder="gt_png/", pred_folder="pred_png/")
# results["All"], results["Things"], results["Stuff"] each give PQ, SQ, RQ.
print(results["All"]["pq"], results["Things"]["pq"], results["Stuff"]["pq"])