Convolution Layers: Channels, Stride, Padding & Dilation

"I take in 64 channels and put out 128. People think that means I doubled something. What I actually did was learn 128 different opinions about the same patch, each one a weighted vote over all 64 inputs."

A Deeply Multi-Channel Convolutional Layer

In Section 19.1 a convolution was one $k \times k$ kernel sliding over one grayscale image. That is enough to argue for the inductive bias, but no useful network looks like that. Color images have three input channels; intermediate feature maps have dozens or hundreds. A layer learns many filters at once, not one. And you constantly need to control the spatial resolution, shrinking it to summarize, holding it to preserve detail, or expanding a filter's reach without paying for more weights. This section adds all of that, ending at the exact PyTorch Conv2d arguments and the shapes that flow through them, the foundation for the network you will train in Section 19.5.

1. Channels: A Filter Spans the Full Input Depth Beginner

The first generalization is depth. A color image is not one $H \times W$ grid but three, stacked: red, green, and blue. In tensor terms it is a $3 \times H \times W$ array, where 3 is the number of input channels. A convolutional filter does not slide three separate $k \times k$ kernels; it slides one $k \times k \times 3$ kernel that reaches across all input channels at once. At each spatial position it multiplies its $k \times k \times 3$ weights against the $k \times k \times 3$ patch beneath it, sums every product into a single number, and that number is one pixel of one output channel. Crucially, the depth of a filter always equals the number of input channels, so you never specify it directly; it is inferred.

A layer learns many such filters, and the count of filters is the number of output channels. Each filter produces its own $H' \times W'$ output map, and these maps stack to form the layer's output of shape $C_{\text{out}} \times H' \times W'$. So a layer mapping a $3 \times 32 \times 32$ color image to a $16 \times 32 \times 32$ feature volume holds $16$ filters, each $3 \times 3 \times 3$, for $16 \times 3 \times 3 \times 3 = 432$ weights plus $16$ biases. Figure 19.2.1 shows the full three-axis structure: depth of each filter set by the input, count of filters set by the desired output channels.

This is the shape you decoded at the end of Section 3.1, where the PyTorch weight tensor was (out_channels, in_channels, kH, kW). Now you know what each axis means: the layer holds out_channels filters, each of depth in_channels, each a $kH \times kW$ spatial grid.

2. The Output-Size Formula Beginner

The spatial output size is governed by one formula you will use constantly. For an input of spatial size $H$ along one axis, a kernel of size $k$, padding $p$ on each side, stride $s$, and dilation $d$, the output size is:

$$ H_{\text{out}} \;=\; \left\lfloor \frac{H + 2p - d\,(k - 1) - 1}{s} \right\rfloor + 1. $$

Each term has a plain meaning. The kernel cannot center on the outermost pixels, so it loses $k - 1$ along each axis (the $d(k-1)$ form generalizes this to dilation, below). Padding $p$ adds $2p$ rows or columns of border, often chosen precisely to cancel that loss. Stride $s$ downsamples the output positions, dividing the count by $s$. The floor handles the case where the kernel does not fit a whole number of times. We will dissect stride, padding, and dilation one at a time, but commit the formula to memory; it predicts every shape mismatch you will ever debug.

3. Stride: Downsampling Inside the Convolution Beginner

Stride is the step size of the sliding window. A stride of 1 visits every pixel; a stride of 2 visits every other pixel, halving the output resolution along each axis. Striding is the convolution's built-in way to downsample: instead of computing a full-resolution map and then shrinking it, you skip positions during the convolution itself, which is cheaper and is the standard way modern architectures reduce spatial size (older networks used pooling, covered in Section 19.3, but strided convolution has largely taken over). With $k = 3$, $p = 1$, $d = 1$, a stride of $s = 2$ on a $32 \times 32$ input gives $\lfloor (32 + 2 - 2 - 1)/2 \rfloor + 1 = 16$, an exact halving.

4. Padding: Controlling Shape and Borders Beginner

Padding adds a border of extra values around the input so the kernel can center on the edge pixels. Without it, every convolution shrinks the map by $k - 1$, and a deep stack would erode a $32 \times 32$ image to nothing after sixteen $3 \times 3$ layers. The common choice is "same" padding, $p = (k-1)/2$ for odd $k$ and stride 1, which makes the output exactly the input size: for $k = 3$ use $p = 1$, for $k = 5$ use $p = 2$. The values placed in the border are usually zeros, though PyTorch also offers reflect and replicate modes, exactly the border strategies you studied in Section 3.6. Zero padding is the default and is rarely worth changing, but it does introduce a faint artifact: pixels near the border see artificial zeros, so the network can in principle read absolute position from the border, partially breaking the position-independence that Section 19.1 prized.

The code below ties the formula to the API. It builds three layers that differ only in stride and padding, and prints the resulting shapes so you can match each to the formula.

import torch
import torch.nn as nn

x = torch.randn(8, 3, 32, 32)        # (batch=8, channels=3, H=32, W=32)

same   = nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1)  # keep 32x32
shrink = nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=0)  # lose 2 -> 30
halve  = nn.Conv2d(3, 16, kernel_size=3, stride=2, padding=1)  # downsample -> 16

print(same(x).shape)    # torch.Size([8, 16, 32, 32])
print(shrink(x).shape)  # torch.Size([8, 16, 30, 30])
print(halve(x).shape)   # torch.Size([8, 16, 16, 16])

# Trainable parameters of one layer: out*in*kH*kW weights + out biases.
print(sum(p.numel() for p in same.parameters()))  # 16*3*3*3 + 16 = 448

5. Dilation: Reach Without Cost Intermediate

Dilation spreads the kernel's sampling points apart, inserting gaps between the weights so a $3 \times 3$ kernel covers a $5 \times 5$ region (dilation 2) or larger, while still using only nine weights. The output-size formula's $d(k-1)$ term captures the enlarged footprint. Dilation buys a larger receptive field (the subject of Section 19.3) at no extra parameters and no extra compute, which is why it became central to semantic segmentation, where you need to see broad context but cannot afford to lose resolution by striding. Figure 19.2.2 contrasts a standard $3 \times 3$ kernel with its dilated counterpart.

6. Two Special Cases: 1x1 and Depthwise Convolution Intermediate

Two degenerate-looking convolutions are workhorses of modern design. The $1 \times 1$ convolution uses a single-pixel kernel, so it does no spatial mixing at all; it is a per-pixel linear combination across channels, exactly a small fully connected layer applied independently at every location. It is the cheapest way to change the channel count, mix information across channels, or add nonlinearity between spatial layers, and it is the backbone of the bottleneck blocks in Chapter 20. The depthwise convolution goes the other way: it convolves each input channel with its own single-channel kernel and does no cross-channel mixing, set in PyTorch by groups=in_channels. The groups argument splits the channels into independent bundles that never see each other; setting it equal to the channel count puts every channel in its own bundle, which is exactly "one private kernel per channel." Pairing a depthwise convolution (spatial mixing, cheap) with a $1 \times 1$ convolution (channel mixing, cheap) is the depthwise-separable convolution that makes MobileNet efficient enough for phones, a topic that returns in Chapter 28.

7. The PyTorch Conv2d API, End to End Intermediate

Everything in this section is one constructor and one forward call. The snippet below builds a two-layer stack that takes a batch of color images and produces a 32-channel feature volume at half resolution, exercising channels, stride, padding, and a $1 \times 1$ projection, and it prints the shape after each layer so you can verify the formula at every step.

import torch
import torch.nn as nn

x = torch.randn(4, 3, 64, 64)        # 4 color images, 64x64

block = nn.Sequential(
    nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1),   # 3 -> 16, keep 64x64
    nn.ReLU(inplace=True),
    nn.Conv2d(16, 32, kernel_size=3, stride=2, padding=1),  # 16 -> 32, halve to 32x32
    nn.ReLU(inplace=True),
    nn.Conv2d(32, 32, kernel_size=1),                       # 1x1 channel mix, keep shape
)

# Trace shapes layer by layer.
h = x
for layer in block:
    h = layer(h)
    if isinstance(layer, nn.Conv2d):
        print(type(layer).__name__, tuple(h.shape))
# Conv2d (4, 16, 64, 64)
# Conv2d (4, 32, 32, 32)
# Conv2d (4, 32, 32, 32)

print("total params:", sum(p.numel() for p in block.parameters()))
# total params: 6432

8. The Backward Pass: Gradients Through a Convolution Advanced

The forward pass is only half of a layer. Training needs the gradients of the loss $L$ with respect to the kernel $W$ (to update it) and with respect to the input $x$ (to pass back to the previous layer). Both fall out of the chain rule applied to the same sum that defines the forward convolution, $y = x * W$, and both turn out to be convolutions themselves, which is why a convolutional layer is so efficient to train. Writing $\delta = \partial L / \partial y$ for the gradient arriving from above, the two gradients are:

The weight gradient is the input cross-correlated with the upstream gradient: each kernel weight touched every spatial position on the forward pass, so its gradient accumulates $\delta$ over all of them. The input gradient is a full (zero-padded) convolution of the upstream gradient with the kernel flipped $180^\circ$, the discrete adjoint of the forward operator, which is exactly the "transposed convolution" used for learnable upsampling in Chapter 24. Both operations reuse the forward machinery, so a framework implements the backward pass with the same fast kernels (im2col, Winograd, or cuDNN) it uses for the forward pass; you never write either by hand, but knowing they are convolutions explains why convolutional training costs about the same as inference per layer.

You now have the complete vocabulary of a convolutional layer: channels in and out, kernel size, stride, padding, and dilation, plus the special $1 \times 1$ and depthwise cases. The one thing the formula hints at but does not explain is how far a deep network can ultimately see. Section 19.3 answers that with the receptive field, and adds pooling, the other classic way to summarize and downsample a feature map.