Why Convolution? Locality, Weight Sharing & Inductive Bias

"A fully connected layer asked me where the cat was. I said: which cat, in which corner, on which day? It had to memorize all of them. I just learn what a cat looks like, once, and check everywhere."

A Smugly Parameter-Efficient Convolutional Layer

In Chapter 18 a network treated its input as a flat vector. For a vector of stock prices or a row of survey responses, that is exactly right: there is no meaningful "neighbor" relationship, and every input feature deserves its own independent weight. For an image, flattening is a small catastrophe. It discards the grid structure that makes a pixel's neighbors informative, and it forces the network to learn the appearance of every pattern separately at every possible location. This section explains why that fails, and how two simple constraints, motivated by the convolution you already built by hand in Chapter 3, fix it. The illustration below previews the whole idea in one picture.

1. The Fully Connected Layer Does Not Scale to Images Beginner

Consider a single modestly sized color image, $224 \times 224$ pixels with 3 channels. Flattened, that is $224 \times 224 \times 3 = 150{,}528$ input features. Connect it to a first hidden layer of just 1000 units, the kind of width that is unremarkable in Chapter 18, and the weight matrix alone has $150{,}528 \times 1000 \approx 1.5 \times 10^{8}$ parameters. One layer, 150 million weights, before you have learned anything. A network deep enough to be useful would have billions of parameters in its first few layers, would need an enormous dataset to constrain them, and would overfit ferociously on anything smaller.

The parameter count is only the visible symptom. The deeper problem is that a fully connected layer has no notion of where a feature is, so it cannot transfer knowledge across positions. Suppose the network learns, in the weights feeding one hidden unit, to detect a vertical edge in the top-left corner. Those weights are useless for detecting the same vertical edge in the bottom-right corner; an entirely separate set of weights, feeding a different hidden unit, must learn the identical pattern from scratch. The network spends its capacity re-deriving the same handful of visual primitives once per location. Figure 19.1.1 contrasts the dense wiring of a fully connected layer with the sparse, repeated wiring of a convolutional one.

2. Constraint One: Local Connectivity Beginner

The first constraint is local connectivity: an output unit connects only to a small spatial neighborhood of the input, not to all of it. This is justified by a fact about images: the pixels that tell you whether there is an edge at position $(x, y)$ are the pixels near $(x, y)$. A pixel a hundred rows away contributes almost nothing to the local question "is there a vertical edge here?". So a hidden unit responsible for a local question needs only local inputs. If the input has spatial size $H \times W$ and each output looks at a $k \times k$ patch, the fan-in of an output drops from $H \times W$ to $k \times k$, which for a typical $3 \times 3$ patch is just 9 connections regardless of how large the image is.

Local connectivity alone is the idea behind a locally connected layer: each output position still has its own private $k \times k$ weights, but it ignores everything outside its patch. This already slashes the connection count, but it does not yet share anything across positions, so the network still relearns the vertical-edge detector separately at every location. We need the second constraint to fix that.

3. Constraint Two: Weight Sharing Beginner

The second constraint is weight sharing: instead of giving each output position its own $k \times k$ weights, every position reuses the same $k \times k$ weights. The justification is again a fact about images, called stationarity: the statistics of natural images are roughly the same everywhere. A vertical edge looks like a vertical edge whether it is in the sky or the pavement, so a detector that works in one place should work in all places. Sharing one set of weights across all positions encodes that belief directly.

The moment we share one $k \times k$ filter across all positions, the operation becomes exactly the cross-correlation you implemented in Section 3.1: slide the kernel, multiply, sum. The layer's entire parameter set is the $k \times k$ kernel (plus one bias), no matter how large the image. The combination of constraints is the layer's name: locally connected with shared weights equals convolution. Table 19.1.1 makes the parameter savings concrete for one $3 \times 3$ filter on a small image.

The contrast in the final column is the whole story of why CNNs work where dense networks cannot. The fully connected layer needs over a million weights; the convolution needs nine. The convolution will need more in practice, one kernel per output channel as we will see in Section 19.2, but the count is governed by the number of distinct features the layer learns, not by the size of the image. A 4-megapixel photograph and a thumbnail use the same kernel weights.

4. Equivariance: The Property That Falls Out for Free Intermediate

Weight sharing gives convolution a clean mathematical property called translation equivariance. A function $f$ is equivariant to a transformation $T$ if shifting the input shifts the output the same way: $f(T(x)) = T(f(x))$. For convolution, if you slide the input image two pixels to the right, every activation in the output slides two pixels to the right, unchanged in value. Formally, writing $S_{\delta}$ for a spatial shift by $\delta$ and $\ast$ for convolution with kernel $w$:

$$ (S_{\delta} x) \ast w \;=\; S_{\delta}(x \ast w). $$

This is not the same as invariance, where the output would be unchanged by the shift. Convolution is equivariant: the features move with the object. Invariance is something you build later, by pooling or by global averaging, when you want a whole-image label that does not care where the object sits. The distinction matters: for classification you eventually want invariance (a cat anywhere is a cat), but for detection in Chapter 23 and segmentation in Chapter 24 you want equivariance preserved, because you need to report where the object is. Convolution gives you equivariance by default and lets you trade it for invariance deliberately.

The code below demonstrates equivariance empirically. We run a fixed convolution on an image, then on a shifted copy of the same image, and confirm that the second output is the shifted first output (away from the borders, where padding breaks the symmetry).

import torch
import torch.nn.functional as F

torch.manual_seed(0)
x = torch.randn(1, 1, 16, 16)                 # (batch, channel, H, W)
w = torch.randn(1, 1, 3, 3)                   # one 3x3 filter

# Shift the input 2 columns to the right (roll, then zero the wrapped edge).
x_shifted = torch.roll(x, shifts=2, dims=3)
x_shifted[..., :2] = 0.0

y        = F.conv2d(x,         w, padding=1)  # cross-correlation, same size
y_shifted = F.conv2d(x_shifted, w, padding=1)

# The output of the shifted input equals the shifted output, in the interior.
expected = torch.roll(y, shifts=2, dims=3)
print(torch.allclose(y_shifted[..., 3:-1], expected[..., 3:-1], atol=1e-5))
# Expected output: True   (equivariance holds away from the border)

This experiment is the runtime echo of the shift-invariance discussion in Section 3.1, where you saw that a designed filter applies identical weights at every location. The only thing Section 19.2 adds is making those weights learnable; the equivariance is inherited from the sliding-window structure, not from any property of the weight values.

Fun Note: The Lazy Genius of Sharing

A fully connected layer is the student who memorizes every worked example in the textbook and panics when the exam reorders the numbers. The convolution is the student who learned the one rule and applies it anywhere. The punchline is that the lazy-looking one wins: by refusing to learn the same edge detector 50,000 times, the convolution frees up its capacity for the parts that actually vary. Remember the layer's motto, learn it once, look everywhere, and you have the whole section in four words, sketched in the illustration below.

5. Inductive Bias: Why the Right Prior Beats Raw Capacity Intermediate

The deep reason convolution wins is best stated in the language of inductive bias. Every learning algorithm needs assumptions to generalize from finite data to unseen inputs; without any assumptions, fitting the training set tells you nothing about new examples. The set of assumptions a model builds in is its inductive bias. A fully connected network has a very weak bias: it can in principle represent any function, including the convolution, but it has no preference for the position-independent, local functions that images actually demand, so it must discover that structure from data, which takes enormous amounts of data.

A convolutional network bakes the structure in. Its hypothesis space contains only local, weight-shared functions, which is a tiny corner of the space of all functions, but it is the corner that contains the good image models. Restricting the hypothesis space is a double-edged tool: if the assumption is right, you generalize far better from the same data because you are not wasting capacity on functions that cannot be the answer; if the assumption is wrong, you cannot fit the truth at all. For natural images, the convolutional assumption is right often enough that for a decade it was the only game in town.

6. When the Bias Is Wrong, and What Came After Advanced

The convolutional prior has real limits, and the rest of Part III is partly a story of relaxing them. Locality means a single convolutional layer cannot relate distant parts of an image; you reach long range only by stacking many layers, which Section 19.3 will quantify with the receptive field. Translation equivariance is the only symmetry convolution provides for free; it is not equivariant to rotation or scale, so a network must learn those variations from data or from augmentation, the topic of Chapter 21. And stationarity is only approximately true: the top of a photograph is statistically more likely to be sky than the bottom, a position-dependent fact that pure weight sharing cannot exploit.

With the argument for convolution settled, the next section makes it real. Section 19.2 takes the single-channel, single-filter operation of this section and equips it with the dimensions a working network needs: many input channels, many output channels, stride, padding, and dilation, along with the exact tensor shapes PyTorch expects.