Data Augmentation: From Flips to MixUp & CutMix

"You showed me the same cat ten thousand times, but flipped, cropped, recolored, and once spliced halfway into a dog. I no longer trust that a cat lives in the left half of the image, and I am better for it."

A Classifier That Stopped Memorizing Backgrounds

The previous section established that the dataset is fixed and finite. Augmentation is how we make a finite dataset behave like a much larger one. The intuition is simple: a horizontally flipped cat is still a cat, so showing the network both versions teaches it that "cat-ness" does not depend on left-right orientation. Each transform we add encodes a piece of knowledge about which variations are irrelevant to the label. Done well, this is the difference between a model that memorizes its training set and one that generalizes; done carelessly, it teaches the model something false. We build the ladder one rung at a time.

1. The Geometric and Photometric Rung Beginner

The first family of augmentations reuses the warps and point operations you already know. The geometric transforms come straight from Chapter 5: horizontal flip, random crop and resize, small rotations, and mild affine shears. Each tells the network that the object's identity survives that transformation. The photometric transforms come from the point operations of Chapter 2: brightness, contrast, saturation, and hue jitter, plus Gaussian blur and added noise. These teach invariance to lighting and camera variation. The standard ImageNet training augmentation is just two of these (a random-resized-crop and a horizontal flip), and that pair alone is worth several points of accuracy.

from torchvision.transforms import v2
import torch

# The classic ImageNet training transform: random crop + flip, then normalize.
train_tf = v2.Compose([
    v2.RandomResizedCrop(224, scale=(0.08, 1.0)),   # zoom + reframe
    v2.RandomHorizontalFlip(p=0.5),                 # left-right invariance
    v2.ColorJitter(brightness=0.2, contrast=0.2,    # lighting invariance
                   saturation=0.2, hue=0.05),
    v2.ToDtype(torch.float32, scale=True),          # uint8 -> [0,1] float
    v2.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])

# At validation time we DO NOT augment: deterministic resize + center crop.
val_tf = v2.Compose([
    v2.Resize(232), v2.CenterCrop(224),
    v2.ToDtype(torch.float32, scale=True),
    v2.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])

2. The Rule That Governs Everything: Respect the Label Beginner

Every augmentation is a bet that the transformation does not change the correct answer, and the bet is task-dependent. A horizontal flip is safe for "is this a cat?" but catastrophic for "which way is this arrow pointing?" or for reading text, where left-right orientation is the label. A vertical flip is fine for satellite imagery but absurd for street scenes, where the sky belongs on top. Aggressive color jitter is harmless for object recognition but destroys a task that depends on color, like grading the ripeness of fruit or classifying a traffic light. The single most important habit in augmentation is to ask, for each transform, "does this preserve the label for my specific task?"

The rule has a sharper edge for tasks with spatial labels. In detection and segmentation, a geometric transform applied to the image must be applied identically to the boxes or masks, or the supervision becomes wrong. Flip the image and the bounding boxes must flip too. This is why production augmentation libraries treat the image and its annotations as a single coupled object, a point the library shortcut at the end of this section makes concrete and which matters greatly for the detection pipelines of Chapter 23 and the segmentation masks of Chapter 24.

3. Cutout, MixUp, and CutMix: Mixing the Labels Intermediate

The next rung of the ladder breaks the assumption that one training image carries one clean label. Three closely related methods, illustrated in Figure 21.2.1, push augmentation into the label space itself. Cutout masks a random rectangle of the image to zero, forcing the network to use the whole object rather than a single discriminative patch. MixUp takes two images and blends them pixel-wise, $\tilde{x} = \lambda x_i + (1 - \lambda) x_j$, and blends their one-hot labels identically, $\tilde{y} = \lambda y_i + (1 - \lambda) y_j$, with $\lambda$ drawn from a Beta distribution (a probability distribution on the interval $[0, 1]$ whose shape parameter controls whether the draws cluster near the ends, giving mostly one image, or near the middle, giving even blends). CutMix combines the two ideas: it pastes a rectangular patch from image $j$ into image $i$, and sets the label mix proportional to the patch's area fraction.

Why does blending labels help? Two reasons. First, it strongly regularizes: the network can no longer be fully confident on any single class, which discourages the overconfident, sharp decision boundaries that overfit. Second, it improves calibration, the agreement between a model's confidence and its actual accuracy, because soft targets teach the model to output soft probabilities. MixUp and CutMix are now standard in competitive image-classification recipes, including the ConvNeXt training procedure from Chapter 20. The implementation is remarkably short, and the illustration below makes the central idea vivid: when you blend two images, you blend their labels too.

import numpy as np
import torch

def mixup(x, y, num_classes, alpha=0.2):
    """MixUp: blend a batch with a shuffled copy of itself."""
    lam = np.random.beta(alpha, alpha)             # small alpha (0.2) -> lam near 0 or 1, mostly one image
    idx = torch.randperm(x.size(0))                # random pairing within batch
    mixed_x = lam * x + (1 - lam) * x[idx]
    y1 = torch.nn.functional.one_hot(y, num_classes).float()
    mixed_y = lam * y1 + (1 - lam) * y1[idx]       # blend the labels too
    return mixed_x, mixed_y

# Train with a loss that accepts soft targets (KL or soft cross-entropy):
# logits = model(mixed_x)
# loss = -(mixed_y * logits.log_softmax(dim=1)).sum(dim=1).mean()

4. Automated Policies: RandAugment and TrivialAugment Intermediate

Once you have a dozen possible transforms, each with a magnitude knob, the search space of "which ones, how strong, in what order" is enormous. The first serious attempt, AutoAugment, used reinforcement learning to search for an optimal policy, but the search itself cost thousands of GPU hours. The field then discovered that almost all of that expense was unnecessary. RandAugment collapses the policy to two interpretable numbers: $N$, how many random transforms to apply per image, and $M$, a single global magnitude that scales all of them. TrivialAugment goes further and removes even those: it applies exactly one randomly chosen transform at a randomly chosen magnitude per image, with zero tuning, and matches or beats the heavily-searched policies.

from torchvision.transforms import v2
import torch

train_tf = v2.Compose([
    v2.RandomResizedCrop(224, antialias=True),
    v2.RandomHorizontalFlip(),
    v2.RandAugment(num_ops=2, magnitude=9),         # the whole policy in one line
    v2.ToDtype(torch.float32, scale=True),
    v2.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])

# MixUp / CutMix operate on the batch (collated tensor + integer labels):
cutmix_or_mixup = v2.RandomChoice([
    v2.MixUp(num_classes=1000),
    v2.CutMix(num_classes=1000),
])
# for images, labels in loader:  images, labels = cutmix_or_mixup(images, labels)