Pooling, Receptive Fields & Feature Hierarchies

"I summarize. You hand me a four-by-four neighborhood and ask what is there; I hand back the single loudest pixel and walk off. The details you lost were details you did not need."

A Max-Pooling Layer With Strong Opinions

Section 19.2 gave you the convolutional layer and the output-size formula, but left one question dangling: a $3 \times 3$ kernel touches only nine input pixels, so how can a CNN ever recognize an object that spans hundreds of pixels? The answer is depth. Each layer reads from the layer below, whose every activation already summarized a patch of the layer below that, and so on down to the pixels. Tracking how far back into the input a given activation reaches is the job of the receptive field, the central quantity of this section. Along the way we add pooling, the older sibling of strided convolution, and end with the feature hierarchy that ties the whole chapter together.

1. Pooling: Summarize and Shrink Beginner

A pooling layer replaces each small window of a feature map with a single summary value, with no learnable weights. The two standard summaries are the maximum (max pooling) and the mean (average pooling). A $2 \times 2$ max pool with stride 2 takes each non-overlapping $2 \times 2$ block and keeps only its largest activation, halving the resolution along each axis and discarding three quarters of the values. Pooling has two purposes: it downsamples, reducing the compute and memory of later layers, and it builds a small amount of local translation invariance, because the maximum over a window does not change if the peak shifts by one pixel within that window. Figure 19.3.1 shows a $2 \times 2$ max pool side by side with average pooling on the same input.

Max pooling dominated early architectures because the strongest response in a window is usually the most informative ("is the edge present somewhere here?"), while average pooling blurs and can dilute a sharp detection. Modern networks frequently skip pooling entirely in favor of strided convolution from Section 19.2, which downsamples and learns its summary jointly. But pooling never disappeared: global average pooling, which collapses an entire $H \times W$ feature map to one number per channel, is the standard final step before a classifier head, because it produces a fixed-length vector regardless of input size and gives the whole-image translation invariance that Section 19.1 distinguished from equivariance.

2. The Receptive Field Intermediate

The receptive field of an activation is the set of input pixels that can influence its value. For the first convolutional layer it is just the kernel size; a $3 \times 3$ filter has a $3 \times 3$ receptive field. The interesting case is deeper layers, where the receptive field grows because each activation reads a window of the previous layer, and each of those activations already had its own receptive field below. The growth follows a clean recurrence. Writing $r_{\ell}$ for the receptive field of layer $\ell$, $k_{\ell}$ for its kernel size, and $j_{\ell-1}$ for the jump (the input-pixel spacing between adjacent activations entering layer $\ell$, equal to the product of all strides up to layer $\ell - 1$):

$$ r_{\ell} \;=\; r_{\ell-1} \;+\; (k_{\ell} - 1)\, j_{\ell-1}, \qquad j_{\ell} \;=\; j_{\ell-1}\, s_{\ell}, $$

starting from $r_0 = 1$ and $j_0 = 1$. To see why the jump matters, picture two side-by-side activations in some deep layer: if every layer below used stride 1 they map to adjacent input pixels (jump 1), but each stride-2 layer doubles the input-pixel gap between them, so after two stride-2 layers neighbors are 4 pixels apart (jump 4). A wider gap means each step of the kernel reaches across more input, which is why the jump multiplies by each layer's stride, and the receptive field grows by $(k-1)$ times the current jump at each layer. The crucial consequence: stride accelerates receptive-field growth, because once the jump is large, each new layer reaches across more input pixels. This is why striding (or pooling) is not just about saving compute; it is the main lever for seeing far.

The code below walks the recurrence over a small stack and prints the receptive field after each layer, so you can see it balloon once stride enters.

def receptive_field(layers):
    """layers: list of (kernel_size, stride). Returns RF after each layer."""
    r, j = 1, 1                       # start: one pixel sees itself, jump 1
    out = []
    for k, s in layers:
        r = r + (k - 1) * j           # grow RF by (k-1)*current_jump
        j = j * s                     # jump scales by this layer's stride
        out.append((r, j))
    return out

# A small VGG-style stack: 3x3 convs with a stride-2 downsample every few layers.
stack = [(3, 1), (3, 1), (3, 2),     # block 1, then halve
         (3, 1), (3, 1), (3, 2),     # block 2, then halve
         (3, 1), (3, 1), (3, 2)]     # block 3, then halve

for i, (r, j) in enumerate(receptive_field(stack), 1):
    print(f"after layer {i}: receptive field {r}x{r}, jump {j}")
# after layer 1: receptive field 3x3, jump 1
# after layer 2: receptive field 5x5, jump 1
# after layer 3: receptive field 7x7, jump 2
# after layer 4: receptive field 11x11, jump 2
# after layer 5: receptive field 15x15, jump 2
# after layer 6: receptive field 19x19, jump 4
# after layer 7: receptive field 27x27, jump 4
# after layer 8: receptive field 35x35, jump 4
# after layer 9: receptive field 43x43, jump 8

3. Effective Versus Theoretical Receptive Field Advanced

The recurrence computes the theoretical receptive field, the pixels that could influence an activation. The pixels that actually influence it meaningfully form a much smaller region. Luo et al. showed that influence falls off roughly like a Gaussian from the center of the receptive field, so the effective receptive field is far smaller than the nominal one, often only a fraction of its diameter, and it grows only as the square root of the number of layers rather than linearly. The reason is statistical: the center pixel reaches the output through exponentially many paths, the corner pixels through very few, so the corners contribute little even though they are technically connected.

The practical lesson is sobering and useful. A network whose theoretical receptive field covers the whole image may, in effect, attend to only a central blob, which is one reason segmentation and detection architectures deliberately enlarge the effective field with dilation (Section 19.2), with explicit large kernels, or with the global context of attention in Chapter 22. When a CNN inexplicably ignores context that is clearly present in the input, a too-small effective receptive field is a prime suspect.

Fun Note: The Network That Claims It Can See

The receptive field is the network bragging about how far it could see; the effective receptive field is how far it actually bothers to look. The corners of a big receptive field are like the back row of a lecture hall: technically attending, contributing almost nothing. So when you compute a glorious $483 \times 483$ theoretical reach and the model still cannot tell forest from farmland, do not be shocked. Keep this slogan on a sticky note: nominal reach is a promise; the Gaussian decides what it keeps. The illustration below captures the gap.

4. The Feature Hierarchy Intermediate

Putting convolution, nonlinearity, and downsampling together produces the structure that gives the chapter its theme: a feature hierarchy. The first layer, with a small receptive field, can only respond to local patterns, and it reliably learns oriented edges and color blobs, the same primitives the Sobel kernels of Chapter 3 and the oriented Gabor filters of Section 4.6 compute by hand. The second layer reads combinations of first-layer features over a slightly larger field and learns corners, junctions, and simple textures. Deeper layers, with receptive fields spanning a large fraction of the image, compose those into object parts (an eye, a wheel, a doorknob), and the deepest layers respond to whole objects and scenes. Figure 19.3.2 sketches this climb.

This hierarchy is not imposed; it emerges from training. Nobody tells layer one to learn edges, yet it almost always does, because edges are the most useful local primitive and the receptive field at that depth permits nothing more global. The hierarchy is also the foundation of transfer learning in Chapter 21: because early-layer features (edges, textures) are generic across tasks, a network trained on one large dataset can be reused on another by keeping its early layers and retraining only the later, task-specific ones. We will see the learned edge detectors directly in Section 19.6.

You can now reason about a CNN's geometry end to end: how big each layer's output is (the formula of Section 19.2) and how far back into the image it reaches (the recurrence here). One obstacle still stands between this understanding and a working network: deep stacks are hard to optimize, because activation statistics drift as signals pass through many layers. Section 19.4 introduces batch normalization, the technique that tamed that drift and made the deep hierarchies of this section trainable in practice.