Vision Datasets & the ImageNet Legacy

"They asked why I subtract 0.485 from the red channel of every photo before I look at it. I told them a committee in 2009 measured the average redness of a million pictures, and I have simply never been given permission to stop."

A Normalization Layer Honoring an Old Tradition

You have spent three chapters building and arranging networks. Now we turn to what you feed them, and the order is deliberate: in this chapter data comes first because every later decision (augmentation, transfer, schedule) is downstream of it. In the previous chapter you loaded CIFAR-10 and ImageNet-pretrained backbones almost without comment. This section makes that data explicit. We will see why a handful of datasets became the field's shared yardsticks, why the statistics of one of them are wired into nearly every model you will ever use, and how a subtle split or sampling mistake can produce a validation number that looks wonderful and means nothing. The illustration below captures the spirit of the whole chapter: the architecture is fixed, and the recipe you drive it with decides where you finish.

1. The Canonical Datasets Beginner

A small set of datasets recur throughout deep computer vision, partly because they are good and partly because shared benchmarks let the field compare methods at all. Knowing their shape and their intended task tells you immediately whether a given dataset fits your problem and how long training will take. The most important ones span four orders of magnitude in size, from tiny digit images to millions of web photographs.

Table 21.1.1 sketches a clear progression. MNIST and CIFAR are for fast iteration: a model trains in minutes, so they are where you debug a pipeline and test a hypothesis cheaply. ImageNet-1k is the pretraining workhorse, large enough that features learned on it transfer almost everywhere. COCO and ADE20K move beyond a single label per image to the box-, mask-, and pixel-level supervision you will need in Chapter 23 and Chapter 24. The web-scale datasets like LAION power the foundation models of Chapter 25 and the text-to-image systems of Chapter 34. The rule of thumb: prototype on CIFAR, transfer from ImageNet, evaluate on whatever benchmark your task community uses.

2. Why ImageNet Still Governs Your Weights Beginner

ImageNet's challenge ended years ago, yet its fingerprints are on nearly every model you build. The reason is transfer learning, which Section 21.3 covers in full: because features learned on ImageNet's 1000 classes turn out to be broadly useful, almost every backbone in Chapter 20 ships with ImageNet-pretrained weights, and you start there rather than from random. That single fact has a precise and easily-missed consequence: a pretrained model expects its inputs preprocessed exactly the way they were during pretraining, which means resized and center-cropped to the training resolution and normalized with ImageNet's per-channel mean and standard deviation.

Those numbers are not arbitrary. They are the channel-wise statistics of the ImageNet training set, $\mu = (0.485, 0.456, 0.406)$ and $\sigma = (0.229, 0.224, 0.225)$ for the red, green, and blue channels of images scaled to $[0, 1]$. Normalization applies the per-channel transform $x' = (x - \mu) / \sigma$, the same standardization idea you met for histograms in Chapter 2, here applied so that the network sees inputs in the distribution it was trained on. Feed it raw 0-to-255 pixels instead and you hand it data from a completely different range; the result, as the practical example below shows, is often near-random output from an otherwise excellent model.

import torch
from torchvision.io import read_image
from torchvision.models import resnet50, ResNet50_Weights

# The weights object carries the EXACT preprocessing it was trained with.
weights = ResNet50_Weights.IMAGENET1K_V2
model = resnet50(weights=weights).eval()

preprocess = weights.transforms()
print(preprocess)
# Expected (abbreviated):
# ImageClassification(
#     crop_size=[224], resize_size=[232],
#     mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], ...)

img = read_image("cat.jpg")               # uint8 tensor, shape (3, H, W)
batch = preprocess(img).unsqueeze(0)      # resize, crop, scale to [0,1], normalize
with torch.no_grad():
    probs = model(batch).softmax(dim=1)
top = probs.argmax(dim=1).item()
print(weights.meta["categories"][top])   # e.g. "tabby, tabby cat"

3. Splits That Do Not Lie Intermediate

The point of holding out data is to estimate how the model behaves on inputs it has never seen. That estimate is only honest if the held-out data is genuinely independent of training, and the standard three-way split exists to protect that independence. The training set fits the weights, the validation set tunes hyperparameters and triggers early stopping, and the test set is touched exactly once, at the very end, to report the final number. The moment you select a model based on its test score, the test set has informed your choices and is no longer an unbiased estimate; it has quietly become a second validation set.

Figure 21.1.1 shows the three roles and the one-way flow of information that keeps each estimate honest.

For small datasets where a single split is noisy, $k$-fold cross-validation reuses the data: partition into $k$ folds, train $k$ times each holding out one fold for validation, and average. The crucial subtlety, and the source of the most common silent bug, is grouping. If your data has natural groups (multiple photos of the same patient, several frames from the same video, augmented copies of one original), every member of a group must land entirely in one split. A naive random split that scatters a patient's scans across train and test lets the model recognize the patient rather than the disease, producing a beautiful test number that evaporates on a genuinely new patient. The code below contrasts the naive split with a group-aware one.

import numpy as np
from sklearn.model_selection import train_test_split, GroupShuffleSplit

# Suppose each row is one X-ray; patient_id groups multiple scans per person.
X = np.arange(1000)
patient_id = np.repeat(np.arange(200), 5)   # 200 patients, 5 scans each

# WRONG: scatters a patient's scans across train and test (leakage).
tr, te = train_test_split(X, test_size=0.2, random_state=0)

# RIGHT: keeps every scan of a patient in exactly one split.
gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=0)
tr_idx, te_idx = next(gss.split(X, groups=patient_id))
overlap = set(patient_id[tr_idx]) & set(patient_id[te_idx])
print("patients shared between train and test:", len(overlap))
# Expected: patients shared between train and test: 0

4. Leakage and Bias: The Silent Result-Wreckers Intermediate

Data leakage is any path by which information from the evaluation set sneaks into training, and it is insidious precisely because it makes your metrics look better. Beyond the grouping leak above, three patterns recur. Augmentation leakage: if you augment first and split second, augmented copies of one image land on both sides; always split, then augment only the training side. Statistic leakage: if you compute normalization statistics over the entire dataset before splitting, the test set has informed the training preprocessing; compute statistics on the training split only. Temporal leakage: for time-ordered data (video, sensor streams), a random split lets the model peek at the future, so split by time instead.

Dataset bias is the quieter cousin of leakage. A model learns whatever correlations are present, including ones you did not intend. The classic cautionary tale is a husky-versus-wolf classifier that achieved high accuracy by detecting snow in the background, because the wolf photos happened to be snowy, not by recognizing the animals at all. ImageNet itself carries geographic and cultural skew (its "groom" and "wedding" images over-represent Western ceremonies), which propagates into every model pretrained on it. The defenses are practical: inspect class balance, view random samples per class, check that the easy-to-vary nuisance factors (lighting, background, camera) are not aligned with the labels, and where possible test on data from a genuinely different source than you trained on.

from torchvision import transforms, datasets
from torch.utils.data import DataLoader, random_split

tf = transforms.Compose([
    transforms.Resize(232), transforms.CenterCrop(224),
    transforms.ToTensor(),                                  # scales to [0,1]
    transforms.Normalize([0.485, 0.456, 0.406],
                          [0.229, 0.224, 0.225]),           # ImageNet stats
])
full = datasets.ImageFolder("data/train", transform=tf)     # class = subfolder name
train, val = random_split(full, [0.8, 0.2])                 # split AFTER loading
loader = DataLoader(train, batch_size=64, shuffle=True, num_workers=4)