LeNet & AlexNet: The Breakthrough Years

"I waited fourteen years for the world to build me a GPU and label a million pictures. People call 2012 an overnight success. I call it the slowest sunrise in history."

A Patient Convolutional Network, Finally Trained

You arrive here having built a convolutional network from scratch in Chapter 19 and trained it on CIFAR-10. You know the conv-BN-ReLU block, pooling, and the receptive field. This section steps back to ask where those parts came from, and the answer is a single network published in 1998 that already contained almost all of them.

The interesting historical fact is that the design was right long before it could win. Understanding why it could not win in 1998, and exactly what 2012 changed, teaches the most useful habit in this chapter: treat an architecture as the solution to a named bottleneck, not as a list of layers to memorize. The illustration below pictures that relay of fixes across a decade, each design dropping the one weight that held the last one back.

1. LeNet-5: The Template That Waited Beginner

LeNet-5, from Yann LeCun and colleagues at Bell Labs, was built to read handwritten digits on bank checks and postal envelopes. It is tiny by modern standards, around 60,000 parameters, yet it established the architectural grammar that every network in this chapter speaks. The flow is: a convolution layer extracts local features, a subsampling (pooling) layer shrinks the spatial map and adds a little translation tolerance, and this pair repeats, doubling the channel count as the spatial size halves, until a few fully connected layers map the final feature vector to class scores. That pattern, spatial size down, channel count up, then classify, is the skeleton of LeNet, AlexNet, VGG, and ResNet alike.

Figure 20.1.1 lays out the LeNet-5 pipeline so you can see the alternation of convolution and subsampling that the rest of the chapter inherits.

We can rebuild LeNet-5 almost verbatim with the layers from Chapter 19. The original used a tanh nonlinearity and an average-pooling variant; the version below substitutes ReLU and max-pooling, which is how the network is taught today, and which trains noticeably faster.

# Rebuild the 1998 LeNet-5 topology with modern layers from Chapter 19.
# Two conv-then-pool stages feed three fully connected layers; the only
# departures from the original are ReLU (for tanh) and max-pool (for avg-pool).
import torch
import torch.nn as nn

class LeNet5(nn.Module):
    """Modern reading of LeNet-5: ReLU + max-pool instead of tanh + avg-pool."""
    def __init__(self, num_classes: int = 10):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 6, kernel_size=5),   # 32x32 -> 28x28, 6 feature maps
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2),                  # 28x28 -> 14x14
            nn.Conv2d(6, 16, kernel_size=5),  # 14x14 -> 10x10
            nn.ReLU(inplace=True),
            nn.MaxPool2d(2),                  # 10x10 -> 5x5
        )
        self.classifier = nn.Sequential(
            nn.Flatten(),                     # 16 * 5 * 5 = 400 features
            nn.Linear(16 * 5 * 5, 120), nn.ReLU(inplace=True),
            nn.Linear(120, 84), nn.ReLU(inplace=True),
            nn.Linear(84, num_classes),
        )

    def forward(self, x):
        return self.classifier(self.features(x))

net = LeNet5()
n_params = sum(p.numel() for p in net.parameters())
print(f"LeNet-5 parameters: {n_params:,}")
y = net(torch.randn(1, 1, 32, 32))
print("output shape:", y.shape)

2. The 2012 ImageNet Moment Beginner

The benchmark that mattered was ImageNet: 1.2 million training images across 1000 classes, scored by top-5 error (the fraction of images whose true label is not among the model's five highest-scoring guesses). Through 2011, the leaderboard was dominated by hand-engineered features (the SIFT and bag-of-visual-words pipelines of Chapter 10 and Chapter 16) feeding linear classifiers, with the best systems near 26% top-5 error. In 2012, AlexNet, a deep convolutional network trained on two GPUs, scored 15.3% top-5 error. A gap of more than ten percentage points over the best classical pipeline was not an incremental win; it was a regime change, and within twelve months almost every serious entry was a deep CNN.

3. The Four Bottlenecks AlexNet Removed Intermediate

AlexNet is best understood not as a new idea but as four targeted fixes layered onto the LeNet template, scaled up to roughly 60 million parameters. Each fix addresses a specific reason a deep network of the time would fail to train or generalize.

Bottleneck one, dead gradients: ReLU. LeNet used tanh, whose gradient saturates toward zero for large positive or negative inputs. Stack several saturating layers and the backpropagated gradient (Chapter 18) shrinks toward nothing, so early layers barely learn. The rectified linear unit $\text{ReLU}(x) = \max(0, x)$ has gradient exactly $1$ for all positive inputs, so it does not saturate on that side. AlexNet reported that ReLU reached a target training error in several times fewer training epochs than tanh on the same network. This is the change that makes deep training practical.

Bottleneck two, compute: GPUs. A network this size was infeasible on CPUs of the era. The authors wrote custom GPU convolution kernels and trained for about a week on two cards. The takeaway generalizes: hardware that makes the forward and backward pass cheap is itself an architectural enabler, a theme that returns when we count FLOPs (floating-point operations, the multiply-adds per image) in Section 20.6.

Bottleneck three, overfitting: dropout and augmentation. Sixty million parameters on 1.2 million images is a recipe for memorization. AlexNet used dropout in the fully connected layers (randomly zeroing half the activations during training, forcing redundant representations) and heavy data augmentation (random crops, horizontal flips, and color jitter, the geometric transforms of Chapter 5 turned into a regularizer). You will study these systematically in Chapter 21.

Bottleneck four, brittle pooling: overlapping max-pool. AlexNet used $3 \times 3$ pooling with stride $2$, so successive pooling windows overlap. This small change reduced error slightly and made the features marginally harder to overfit. The implementation below assembles all four fixes.

# AlexNet assembles the four bottleneck fixes onto the LeNet template:
# ReLU after every conv, an overlapping 3x3 stride-2 max-pool, and dropout
# in the dense head. The wide 11x11 stride-4 stem shrinks 224x224 fast.
import torch
import torch.nn as nn

class AlexNet(nn.Module):
    """AlexNet for 224x224 ImageNet inputs (the torchvision topology)."""
    def __init__(self, num_classes: int = 1000):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),  # large 11x11 stem
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),                  # overlapping pool
            nn.Conv2d(64, 192, kernel_size=5, padding=2), nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(192, 384, kernel_size=3, padding=1), nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1), nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1), nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        self.avgpool = nn.AdaptiveAvgPool2d((6, 6))
        self.classifier = nn.Sequential(
            nn.Dropout(0.5), nn.Linear(256 * 6 * 6, 4096), nn.ReLU(inplace=True),
            nn.Dropout(0.5), nn.Linear(4096, 4096), nn.ReLU(inplace=True),
            nn.Linear(4096, num_classes),
        )

    def forward(self, x):
        x = self.avgpool(self.features(x))
        return self.classifier(torch.flatten(x, 1))

net = AlexNet()
print(f"AlexNet parameters: {sum(p.numel() for p in net.parameters()):,}")
print(net(torch.randn(2, 3, 224, 224)).shape)

Notice where the parameters live. The convolutional layers carry only a few million weights, while the first fully connected layer alone (mapping $256 \times 6 \times 6 = 9216$ activations to $4096$) holds about 37 million. This imbalance, cheap convolutions and expensive dense layers, is the inefficiency that VGG inherits and that the $1 \times 1$ bottleneck of Inception and global average pooling later remove. The seed of Section 20.2 is already visible here.

# Load AlexNet with ImageNet-pretrained weights and the matching preprocessing.
# weights.transforms() is the exact resize/crop/normalize the weights expect,
# so you never reconstruct the input pipeline by hand.
from torchvision.models import alexnet, AlexNet_Weights
weights = AlexNet_Weights.IMAGENET1K_V1
model = alexnet(weights=weights).eval()
preprocess = weights.transforms()  # resize, center-crop, normalize, all handled
# model(preprocess(pil_image).unsqueeze(0)) -> 1000 ImageNet logits

4. Why It Worked: Scale Met a Ready Idea Intermediate

It is tempting to read AlexNet as a clever new architecture, but the more accurate and more useful reading is that a fourteen-year-old idea finally met the three things it had always needed: enough labeled data (ImageNet), enough compute (GPUs), and a nonlinearity that did not strangle gradients in a deep stack (ReLU). The convolutional structure was essentially LeNet's. This reframing matters because it tells you what to look for in every subsequent architecture: not "what novel layer did they invent?" but "what specific ceiling did they raise, and how would I detect that I am hitting the same ceiling in my own model?"