Smoothing: Box, Gaussian & Median Filters

"My therapist says I average things out instead of confronting them. I told her that with enough neighbors, every problem regresses to the mean."

An Exceedingly Mellow Gaussian Filter

The previous section built the sliding-window machinery and showed that the kernel weights determine the behavior. This section commits to the first and most common choice of weights: all of them positive, summing to one. Such kernels compute weighted averages, and weighted averages smooth. The question that separates a good smoother from a bad one is subtler than it looks: which average, over which neighborhood, and what happens to the parts of the image that are not noise?

1. Why Averaging Works: The Statistics of Noise Beginner

Recall from Chapter 1 where image noise comes from: photon shot noise, sensor read noise, and quantization, which together make each recorded pixel a noisy measurement of the true scene radiance. Model a pixel as $I(x,y) = s(x,y) + n(x,y)$, where $s$ is the clean signal and $n$ is zero-mean noise with standard deviation $\sigma_n$, independent across pixels. Averaging $N$ such measurements leaves the signal untouched (the average of $s$ is $s$, if $s$ is locally constant) while the noise standard deviation falls to $\sigma_n / \sqrt{N}$. A $3 \times 3$ average is nine measurements: noise drops by a factor of three. A $5 \times 5$ average drops it by five.

That argument contains both the promise and the catch. The promise: noise reduction scales with the square root of the neighborhood size, for free, with nine numbers. The catch sits in the clause "if $s$ is locally constant." Real images are locally constant almost everywhere, but the exceptions, edges and fine texture, are precisely the parts that carry most of the visual information. Averaging across an edge mixes the two sides and produces blur. Every smoother in this section makes some version of this trade, and managing the trade is the art of denoising.

2. The Box Filter Beginner

The box filter (or mean filter) is the uniform average: every weight in a $k \times k$ kernel equals $1/k^2$. It is the filter we used to introduce correlation in Section 3.1, it is the cheapest smoother in existence (Section 3.6 shows it can run in constant time per pixel regardless of $k$, via running sums), and for casual noise knockdown it is fine. OpenCV exposes it as cv2.blur.

But the box has a defect visible to the naked eye: it treats a pixel at the corner of the window as seriously as the center pixel, so the window's hard cutoff imprints itself on the output. Smoothed point lights become little squares, and repeated box filtering produces streaky, blocky artifacts. In the frequency-domain language of Chapter 4, the box filter's spectrum rings: it fails to suppress some high frequencies while distorting mid frequencies. The fix is to let the weights taper smoothly with distance from the center, which is exactly the Gaussian.

3. The Gaussian Filter Intermediate

The Gaussian filter weights each neighbor by a bell curve over its distance from the center, as the right panel of Figure 3.2.1 shows:

$$ G_\sigma(i, j) \;=\; \frac{1}{2\pi\sigma^2}\, \exp\!\left( -\,\frac{i^2 + j^2}{2\sigma^2} \right) $$

The single parameter $\sigma$, the standard deviation in pixels, sets the smoothing scale: detail much finer than $\sigma$ is averaged away, structure much coarser survives. The kernel size is a separate, practical choice: since the bell decays fast, weights beyond about $3\sigma$ are negligible, and the standard recipe is a kernel spanning $\pm 3\sigma$, that is, size $2\lceil 3\sigma \rceil + 1$. A $\sigma$ of 1.5 wants roughly an $11 \times 11$ kernel; pass ksize=(0, 0) to cv2.GaussianBlur and OpenCV computes a suitable size for you.

Three properties make the Gaussian the default smoother of the field rather than just one option among many. First, it is the unique rotationally symmetric filter that is also separable (it factors into a horizontal pass followed by a vertical pass), which makes large-radius Gaussian smoothing cheap; Section 3.6 quantifies the speedup. Second, it cascades cleanly: blurring with $\sigma_1$ and then $\sigma_2$ equals one blur with $\sigma = \sqrt{\sigma_1^2 + \sigma_2^2}$, the property on which the scale spaces and image pyramids of Chapter 4 are built. Third, it introduces no artifacts: smoothing with a wider Gaussian never creates structure that was not in the image, a guarantee (formalized in scale-space theory) that the box filter conspicuously lacks.

Building the kernel from the formula is a five-line exercise, worth doing once to demystify it:

import numpy as np
import cv2

def gaussian_kernel(size: int, sigma: float) -> np.ndarray:
    """Build a normalized 2D Gaussian kernel of odd size."""
    ax = np.arange(size) - size // 2          # e.g. [-2,-1,0,1,2] for size 5
    xx, yy = np.meshgrid(ax, ax)
    k = np.exp(-(xx**2 + yy**2) / (2.0 * sigma**2))
    return k / k.sum()                         # normalize: weights sum to 1

k = gaussian_kernel(5, sigma=1.0)
print(k.round(3))
# [[0.003 0.013 0.022 0.013 0.003]
#  [0.013 0.06  0.098 0.06  0.013]
#  [0.022 0.098 0.162 0.098 0.022]
#  [0.013 0.06  0.098 0.06  0.013]
#  [0.003 0.013 0.022 0.013 0.003]]
img = cv2.imread("portrait.jpg", cv2.IMREAD_GRAYSCALE)
smooth = cv2.filter2D(img, -1, k)              # apply with Section 3.1 machinery

4. The Median Filter Intermediate

Now consider a different noise regime. Salt-and-pepper noise, from dead pixels, transmission glitches, or extreme sensor events, replaces isolated pixels with values near 0 or 255. Averaging is the wrong response: a single 255 outlier in a $3 \times 3$ window shifts the mean by up to 28 gray levels, smearing the corruption across the neighborhood instead of removing it. What you want is an estimator that ignores outliers entirely. Statistics has one ready: the median.

The median filter replaces each pixel with the median of its window's values. As Figure 3.2.2 illustrates, an outlier in the window lands at the end of the sorted order and simply never gets selected; the output is an actual pixel value from the neighborhood majority. Up to half the window can be corrupted before the median breaks. Note what the median filter is not: it is not a weighted average, not linear, not expressible as any kernel. Our impulse-response analysis from Section 3.1 does not apply to it (filter an impulse and you get all zeros back, as the median of a window containing one bright pixel and eight black ones is black). It is the chapter's first genuinely nonlinear filter, the opening move of a theme that continues with the bilateral filter in Section 3.5.

The experiment below corrupts an image with 5 percent salt-and-pepper noise and compares the two repair strategies head to head, scoring with PSNR (the fidelity metric from Chapter 1).

import cv2
import numpy as np

rng = np.random.default_rng(7)
img = cv2.imread("lighthouse.png", cv2.IMREAD_GRAYSCALE)

# Corrupt 5% of pixels: half to 0 (pepper), half to 255 (salt).
noisy = img.copy()
mask = rng.random(img.shape)
noisy[mask < 0.025] = 0
noisy[mask > 0.975] = 255

gauss  = cv2.GaussianBlur(noisy, (5, 5), 1.0)   # linear repair attempt
median = cv2.medianBlur(noisy, 5)               # order-statistic repair

for name, result in [("noisy", noisy), ("gaussian", gauss), ("median", median)]:
    print(f"{name:9s} PSNR = {cv2.PSNR(img, result):5.2f} dB")
# Representative output:
# noisy     PSNR = 15.21 dB
# gaussian  PSNR = 26.88 dB   <- outliers smeared, edges blurred
# median    PSNR = 33.40 dB   <- outliers deleted, edges intact

Two practical notes on cv2.medianBlur. The kernel size must be odd, and for sizes above 5 the input must be 8-bit; OpenCV uses a constant-time histogram-based algorithm for uint8 that keeps large-window medians affordable. And because the median preserves sharp steps while flattening small fluctuations, heavy median filtering produces a distinctive posterized, "painted" look; that property is exploited deliberately by stylization apps, and is a clue that you have over-filtered when it appears by accident.

5. Choosing a Smoother Beginner

Table 3.2.1 condenses the section into a decision aid. The columns reflect the three questions to ask of any smoothing job: what noise am I fighting, how much edge damage can I afford, and how fast must it run?

One pattern from the table is worth promoting to a habit: the Gaussian is the default pre-filter, not necessarily the final denoiser. Nearly every derivative computation in Section 3.4, every detector in Chapter 9, and every pyramid level in Chapter 4 begins with a Gaussian blur whose job is to set the analysis scale and tame noise before a more fragile operation runs. When the smoothed image is itself the product, and its edges matter, the median here and the edge-preserving filters of Section 3.5 take over.