Class Imbalance, Label Noise & Real-World Data

"Ninety-nine times out of a hundred the part was fine, so I learned to say fine and was right ninety-nine percent of the time. They were furious. The whole point, they said, was the hundredth part. Nobody mentioned this during training."

A Defect Detector That Optimized the Wrong Thing

The previous four sections assumed something close to the clean, balanced benchmark world of Section 21.1. Real projects rarely live there. A medical screening set has far more healthy than diseased scans; a manufacturing line produces vastly more good parts than defective ones; a wildlife camera trap captures common species constantly and the endangered one almost never. And in every dataset, some labels are wrong, because annotation is done by tired humans under time pressure. This section is about training well anyway. It connects back to the noise models of Chapter 7, where noise corrupted pixels; here it corrupts labels.

1. The Imbalance Problem and Why Accuracy Lies Beginner

Consider a defect detector where 99% of parts are good. A model that predicts "good" for everything achieves 99% accuracy while being completely useless, because it never catches a single defect, the one thing it exists to do. This is why accuracy is the wrong metric under imbalance. Cross-entropy, the standard classification loss from Section 18.5, has the same blind spot: summed over a batch that is 99% good parts, the loss is dominated by the easy majority class, so the gradient mostly teaches the model to be even more confident about good parts and barely registers the rare defects. The model converges to the trivial majority predictor because that is what the loss rewards. The illustration below dramatizes the trap: a forecaster who always predicts sun looks brilliant on paper and misses the one event that mattered.

The first fix is the metric. Under imbalance you measure precision, recall, and their harmonic mean the F1 score, per class, or you summarize with balanced accuracy (the average of per-class recalls) or the area under the precision-recall curve. These reward catching the rare class. Choosing the right metric is not a cosmetic decision; it determines what "good" even means, and it sets the threshold tuning you will revisit when per-pixel logits get thresholded in the segmentation of Chapter 24. The metric evolution from PSNR in Chapter 1 through intersection over union (IoU) and mean average precision (mAP) runs straight through this choice.

2. Loss-Level Fixes: Class Weights and Focal Loss Intermediate

The simplest loss-level fix is class weighting: multiply each class's contribution to the loss by a weight inversely related to its frequency, so a rare-class mistake costs more than a common-class mistake. A refinement, the class-balanced loss, weights by the "effective number" of samples rather than the raw inverse frequency, which behaves better when counts vary by orders of magnitude. The intuition is that near-duplicate examples in a large class add little new information, so the thousandth photo of a common class counts for less than the raw count suggests. Concretely, a class of $n$ raw samples is assigned an effective number $E_n = (1 - \beta^n) / (1 - \beta)$ with $\beta$ just under $1$ (Cui et al., 2019), and the class weight is set to $1 / E_n$. The point of this particular form is that it saturates: as $n$ grows, $\beta^n \to 0$ and $E_n$ flattens toward the ceiling $1 / (1 - \beta)$, so doubling an already-large class barely changes its weight, while a rare class with small $n$ still gets nearly the full inverse-frequency boost. Weighting by this saturating effective count stops the loss from over-amplifying a handful of rare-class samples.

Class weighting rebalances how much each class counts; focal loss instead rebalances how much each example counts. Focal loss is the more surgical tool, designed for the extreme imbalance of object detection. It multiplies the standard cross-entropy by a factor $(1 - p_t)^\gamma$, where $p_t$ is the model's predicted probability for the true class. Easy examples (high $p_t$) get their loss driven toward zero, so the gradient is dominated by the hard and usually rare cases.

Figure 21.5.1 shows how the focal factor reshapes the loss: well-classified easy examples are suppressed so the model stops spending capacity on what it already knows and focuses on the hard tail.

import torch
import torch.nn.functional as F

def focal_loss(logits, targets, gamma=2.0, alpha=None):
    """Multi-class focal loss: down-weights easy (confident) examples."""
    log_p = F.log_softmax(logits, dim=1)
    log_pt = log_p.gather(1, targets.unsqueeze(1)).squeeze(1)   # log prob of true class
    pt = log_pt.exp()
    loss = -((1 - pt) ** gamma) * log_pt                        # the focal factor
    if alpha is not None:                                       # optional per-class weight
        loss = alpha[targets] * loss
    return loss.mean()

logits = torch.randn(8, 3); targets = torch.randint(0, 3, (8,))
print("CE   :", F.cross_entropy(logits, targets).item())
print("focal:", focal_loss(logits, targets, gamma=2.0).item())
# Representative (unseeded, so values vary):
# CE   : 1.27
# focal: 0.61
# focal loss is smaller because confidently-correct examples contribute almost nothing

3. Data-Level Fixes: Resampling Intermediate

Instead of (or alongside) reweighting the loss, you can rebalance the data the model sees. Oversampling draws rare-class examples more often, so each batch is closer to balanced; PyTorch's WeightedRandomSampler does this cleanly by assigning each sample a draw probability inversely proportional to its class frequency. Undersampling discards majority-class examples to match the minority, which is simpler but throws away data. A middle path that often works best pairs modest oversampling with the augmentation of Section 21.2, so the repeated rare-class examples are not identical copies (which would just be memorized) but varied views. The decoupling insight from recent long-tailed-recognition research is worth knowing: it is often best to train the feature backbone on the natural imbalanced distribution and only rebalance when training the final classifier.

import torch
from torch.utils.data import DataLoader, WeightedRandomSampler

labels = torch.tensor([0]*990 + [1]*10)          # 99:1 imbalance
class_count = torch.bincount(labels).float()
sample_weight = (1.0 / class_count)[labels]      # rare class -> high draw weight
sampler = WeightedRandomSampler(sample_weight,
                                num_samples=len(labels), replacement=True)
loader = DataLoader(dataset, batch_size=64, sampler=sampler)
# Each batch now sees roughly balanced classes despite the 99:1 raw ratio.
# Pair with augmentation so oversampled rare examples are varied, not duplicated.

4. Label Noise: Even ImageNet Is Wrong Sometimes Advanced

Every real dataset contains wrong labels, and so do the benchmarks. As noted in Section 21.1, confident-learning analyses estimate roughly a 6% label-error rate in the ImageNet validation set and similar rates across CIFAR and others, errors large enough to reorder model leaderboards. The danger is that a high-capacity network, given enough epochs, will memorize even random labels, so it happily fits your mistakes. The empirical pattern that makes this tractable is that networks tend to learn the clean, consistent majority pattern first and only memorize the noisy minority later in training. That ordering is the lever for almost every noise-robust method.

Several techniques exploit it. Early stopping halts training before the memorization phase, a free first defense. Robust losses like the generalized cross-entropy or symmetric cross-entropy bound the loss a single mislabeled example can contribute, so one wrong label cannot dominate the gradient. Co-teaching trains two networks that each select the small-loss (probably-clean) examples for the other, since clean examples have low loss in the early-learning phase. Confident learning (the cleanlab approach) estimates which labels are likely wrong from the model's own predictions and lets you remove or relabel them. Label smoothing from Section 21.4 also helps modestly, since softened targets reduce how hard the model chases any single label.

from cleanlab.filter import find_label_issues
import numpy as np

# pred_probs: (N, K) cross-validated predicted probabilities; labels: (N,) given labels
issue_idx = find_label_issues(
    labels=labels, pred_probs=pred_probs,
    return_indices_ranked_by="self_confidence",   # worst-looking labels first
)
print(f"{len(issue_idx)} of {len(labels)} labels flagged as likely errors")
# Inspect the top-ranked issues by eye, then relabel or drop them and retrain.