VGG & Inception: Depth vs Width

"One of us stacked the same small filter until the lights flickered. The other ran four filter sizes at once and let the gradient pick a favorite. We both won the same year. Architecture is not a religion."

Two Networks That Disagreed Productively in 2014

In Section 20.1 we saw AlexNet remove the compute and gradient ceilings, and we noticed that its parameters piled up in the dense layers, not the convolutions. With training now feasible, the 2014 question became architectural: given that you can stack convolutions, how should you arrange them? VGG and Inception gave opposite-flavored answers in the same ImageNet competition, and both finished near the top. Studying them side by side teaches the two most reusable structural ideas in convolutional design.

1. VGG: The Power of Uniform Depth Beginner

VGG, from Oxford's Visual Geometry Group, is almost aggressively simple. Every convolution is $3 \times 3$ with stride $1$ and padding $1$ (so spatial size is preserved), every pooling is $2 \times 2$ max-pool, and the only design knobs are how many convolutions to put between pools and how many channels each stage gets. VGG-16 stacks thirteen such convolutions and three fully connected layers; the depth, not any clever module, is the whole idea. The network is organized into stages, and within a stage the channel count is constant while the spatial size is fixed; pooling between stages halves the spatial size and the channel count doubles, the same spatial-down, channels-up rhythm of LeNet.

The central VGG argument is the substitution of small kernels for large ones. Two stacked $3 \times 3$ convolutions see the same $5 \times 5$ input region as one $5 \times 5$ convolution does, because the receptive field of Chapter 19 grows additively with depth. But the two small layers are cheaper and add an extra nonlinearity. Count the weights for a layer with $C$ input and $C$ output channels (each filter has kernel-area $\times\, C$ weights and there are $C$ of them, so the count is kernel-area $\times\, C^2$): one $5 \times 5$ convolution costs $25 C^2$ parameters, while two $3 \times 3$ convolutions cost $2 \times 9 C^2 = 18 C^2$, about 28% fewer, and three $3 \times 3$ layers cost $27 C^2$ for the same receptive field as one $7 \times 7$ layer at $49 C^2$. Figure 20.2.1 shows this equivalence.

Because every kernel is the same size and every stage follows the same rule, the whole VGG trunk can be generated from a short configuration list rather than written out layer by layer. The code below builds the thirteen-convolution trunk by mapping one block-builder over a list of stage shapes.

# Generate the whole VGG-16 trunk from one repeated rule: a stage is
# n_convs of (3x3 conv, ReLU) followed by a 2x2 max-pool. The uniformity
# is the design, so the network is a short loop over a config list.
import torch.nn as nn

def vgg_block(in_ch, out_ch, n_convs):
    """One VGG stage: n_convs of 3x3 conv-ReLU, then a 2x2 max-pool."""
    layers = []
    for i in range(n_convs):
        layers += [nn.Conv2d(in_ch if i == 0 else out_ch, out_ch,
                             kernel_size=3, padding=1),
                   nn.ReLU(inplace=True)]
    layers.append(nn.MaxPool2d(2))           # halve spatial size between stages
    return nn.Sequential(*layers)

# VGG-16 convolutional stages: (in, out, n_convs)
cfg = [(3, 64, 2), (64, 128, 2), (128, 256, 3), (256, 512, 3), (512, 512, 3)]
features = nn.Sequential(*[vgg_block(i, o, n) for i, o, n in cfg])
n_convs = sum(n for _, _, n in cfg)
print(f"{len(cfg)} stages, {n_convs} convolutions in total")

VGG's weakness is the one AlexNet already exposed: its three fully connected layers hold roughly 120 million of its 138 million parameters, almost all of them in the first dense layer. The convolutional trunk is elegant and cheap; the classifier head is a parameter sink. Inception attacks exactly this.

2. Inception: Several Scales at Once Intermediate

The Inception (GoogLeNet) team made the opposite bet. Rather than commit to one kernel size and go deep, they asked: why choose a kernel size at all? An object can occupy a small or a large fraction of the image, so let one module compute $1 \times 1$, $3 \times 3$, and $5 \times 5$ convolutions in parallel, plus a pooling branch, and concatenate the results along the channel axis. The next layer then has access to features at several scales simultaneously and learns which to weight. This is the multi-scale idea of the image pyramids in Chapter 4, now computed inside a single learnable block. The workshop illustration below makes the parallel-branch idea concrete (the funnels are the cheap channel bottlenecks introduced just below).

The obvious problem is cost: running a $5 \times 5$ convolution on a wide feature map is expensive, and concatenating branches makes the next layer's input very wide. The fix is the section's second reusable primitive. A $1 \times 1$ convolution mixes channels at each spatial location without touching the spatial extent, so it can reduce the channel count cheaply before an expensive $3 \times 3$ or $5 \times 5$ branch. Placing a $1 \times 1$ reduction in front of each expensive branch is the "Inception with dimension reduction" module, shown in Figure 20.2.2.

# One Inception module runs four branches on the same input and concatenates
# them on the channel axis. Each expensive 3x3 and 5x5 branch is fronted by a
# 1x1 reduction so the wide module stays affordable, exactly as in Figure 20.2.2.
import torch
import torch.nn as nn

class Inception(nn.Module):
    """One Inception module: four parallel branches concatenated on channels."""
    def __init__(self, in_ch, c1, c3_red, c3, c5_red, c5, pool_proj):
        super().__init__()
        relu = nn.ReLU(inplace=True)
        self.b1 = nn.Sequential(nn.Conv2d(in_ch, c1, 1), relu)
        self.b2 = nn.Sequential(nn.Conv2d(in_ch, c3_red, 1), relu,        # 1x1 reduce
                                nn.Conv2d(c3_red, c3, 3, padding=1), relu)
        self.b3 = nn.Sequential(nn.Conv2d(in_ch, c5_red, 1), relu,        # 1x1 reduce
                                nn.Conv2d(c5_red, c5, 5, padding=2), relu)
        self.b4 = nn.Sequential(nn.MaxPool2d(3, stride=1, padding=1),
                                nn.Conv2d(in_ch, pool_proj, 1), relu)

    def forward(self, x):
        return torch.cat([self.b1(x), self.b2(x), self.b3(x), self.b4(x)], dim=1)

# The classic "inception_3a" config from GoogLeNet:
mod = Inception(192, 64, 96, 128, 16, 32, 32)
out = mod(torch.randn(1, 192, 28, 28))
print("output channels:", out.shape[1])  # 64 + 128 + 32 + 32

GoogLeNet also replaced VGG's parameter-heavy dense head with global average pooling: average each final feature map down to a single number, then apply one small linear layer. This collapses the spatial map to a length-$C$ vector with zero parameters, eliminating the 100-million-weight dense layers entirely. GoogLeNet reached accuracy comparable to VGG with only about 6.8 million parameters, roughly twenty times fewer than VGG-16 (and about twelve times fewer than AlexNet, the comparison the original paper highlights), a direct payoff of fixing the dense-head bottleneck this chapter has flagged since AlexNet.

3. Depth versus Width as a Living Tradeoff Intermediate

VGG and Inception are the two poles of a tradeoff that never goes away. Depth (more sequential layers) builds abstraction and grows the receptive field, but plain depth eventually stops helping, the very wall that ResNet hits in Section 20.3. Width (more parallel branches or channels) increases representational capacity at each level and parallelizes well on hardware, but costs memory and can waste capacity if the extra width is redundant. Modern designs do not pick a side; they tune both, and EfficientNet (Section 20.4) makes the joint tuning explicit by scaling depth, width, and resolution together with a single coefficient.

# Load VGG-16 and GoogLeNet with ImageNet weights, no block definitions needed.
# Slicing vgg.features to a layer index gives the frozen feature extractor used
# for the perceptual loss of subsection 3.
from torchvision.models import vgg16, VGG16_Weights, googlenet, GoogLeNet_Weights
vgg = vgg16(weights=VGG16_Weights.IMAGENET1K_V1).eval()
goog = googlenet(weights=GoogLeNet_Weights.IMAGENET1K_V1).eval()
# To use VGG as a feature extractor (perceptual loss, subsection 3):
feat = vgg.features[:16]  # truncate to a chosen layer, freeze, done.