Spatial Filtering & Convolution

Chapter Overview

Everything in Chapter 2 shared one limitation: each output pixel depended only on the single input pixel at the same position. Point operations can brighten, stretch, and threshold, but they are blind to structure. They cannot tell a noisy speckle from a fine texture, or an edge from a gradient, because telling those apart requires looking at a pixel's surroundings. This chapter widens the aperture from one pixel to a neighborhood, and in doing so unlocks the operations that define classical image processing: smoothing, sharpening, and differentiation.

The instrument that does all of this is the kernel: a small grid of weights, typically $3 \times 3$ to $31 \times 31$, that slides across the image and computes a weighted sum at every position. Section 3.1 builds this machinery carefully, distinguishing correlation from true convolution (they differ by a flip that matters more in theory than in daily practice) and implementing both from scratch in NumPy before handing the job to OpenCV and PyTorch. The remaining sections are a tour of what different weights buy you. Section 3.2 covers the smoothing family: the box filter, the Gaussian (the most-used filter in vision), and the median, a nonlinear order-statistic filter that linear theory cannot replicate. Section 3.3 runs the logic in reverse and sharpens, using the unsharp mask trick photographers invented in darkrooms a century before Photoshop. Section 3.4 turns filters into measuring instruments: Sobel, Laplacian, and LoG kernels estimate first and second derivatives, producing the gradient maps that feed the edge and line detectors of Chapter 9.

Two closing sections address the tensions the first four create. Smoothing fights noise but destroys edges; Section 3.5 resolves the conflict with the bilateral and guided filters, which adapt their weights to image content and smooth within regions while respecting boundaries. These edge-aware filters run inside virtually every smartphone camera pipeline shipped today. Finally, Section 3.6 confronts the two engineering questions every practitioner hits within an hour of filtering real images: what happens at the image border, where the kernel hangs off the edge, and how to make filtering fast, where one algebraic property (separability) routinely buys a tenfold speedup.

A word on why this chapter deserves unusual attention. The convolution kernel is this book's signature recurring character. In Part I it is designed by hand: you will choose the weights and know exactly why each one is there. In Chapter 19 the same sliding-window operation returns with learnable weights, and remarkably, the first layers of trained networks rediscover the very filters you build here: oriented edge detectors, blob detectors, and color-opponent kernels. In Chapter 33, stacks of convolutions form the U-Net that turns noise into photographs. Understanding what a $3 \times 3$ kernel can and cannot see is therefore not classical trivia; it is the foundation for reading every architecture diagram in the second half of this book.