Background Subtraction & Change Detection

"I have stared at this hallway for nine thousand frames. I know exactly what nothing looks like. The instant something is not nothing, I will let you know."

A Patient Background Model With Excellent Object Permanence

In the previous three sections the camera and the scene were both allowed to move, and we worked hard to estimate that motion densely or sparsely. This section makes one strong assumption, that the camera is fixed, and is rewarded with an enormous simplification. Under a static camera, the projected position of every static 3D point is constant: the floor, the walls, the parked cars never move in the image. Anything that does move, a person, a vehicle, a falling box, lights up as a difference from the remembered scene. We never compute a velocity; we compute a membership label, foreground or background, at every pixel of every frame. This is change detection, and background subtraction is its dominant classical form.

1. The Static-Camera Assumption and the Simplest Detector Beginner

Start with the most naive idea that could possibly work: keep one reference image $B(x, y)$ of the empty scene, and for each new frame $I_t$, mark a pixel as foreground when it differs from the reference by more than a threshold $\tau$:

$$ F_t(x, y) = \begin{cases} 1 & \text{if } \lvert I_t(x, y) - B(x, y) \rvert > \tau \\ 0 & \text{otherwise.} \end{cases} $$

This is frame differencing against a fixed reference, and it has exactly one virtue and many flaws. The virtue is speed: one subtraction and one comparison per pixel. The flaws are that you need a clean empty frame to begin with (rare in practice), and that the reference goes stale the moment a cloud passes, a light switches, or the sun moves. The fix for staleness is to let the background adapt. The simplest adaptive model is a running average: blend a little of each new frame into the background so it slowly tracks lighting changes while ignoring transient foreground.

$$ B_t = (1 - \alpha) B_{t-1} + \alpha I_t, \qquad 0 < \alpha \ll 1. $$

The learning rate $\alpha$ controls the model's memory: small $\alpha$ (say $0.01$) means a slow, stable background that needs roughly $1/\alpha$ frames to forget the past, while large $\alpha$ lets fast lighting changes in but also bakes a slow-moving foreground object into the background, where it leaves a ghost. Code Fragment 1 implements the running-average detector and shows the staleness-versus-stability tradeoff in one parameter.

import cv2
import numpy as np

cap = cv2.VideoCapture("hallway.mp4")
ok, frame = cap.read()
bg = frame.astype(np.float32)        # running-average background, kept in float
alpha = 0.02                         # learning rate: ~50-frame memory

while True:
    ok, frame = cap.read()
    if not ok:
        break
    cur = frame.astype(np.float32)
    # per-pixel absolute difference, collapsed across color channels
    diff = cv2.absdiff(cur, bg).max(axis=2)
    fg = (diff > 25).astype(np.uint8) * 255          # tau = 25 gray levels

    # update the background ONLY where we believe it is background,
    # so moving objects are not blended in and do not leave ghosts
    mask = (fg == 0)[..., None]
    bg = np.where(mask, (1 - alpha) * bg + alpha * cur, bg)

    cv2.imshow("foreground", fg)
    if cv2.waitKey(1) == 27:
        break
# Typical result: clean silhouettes of walkers; the floor stays black,
# and the model re-absorbs a bag left on the floor over ~50 frames.

The masked update in Code Fragment 1 is the difference between a toy and something usable. If you update the background everywhere, a person who pauses for a few seconds gets absorbed into the model and vanishes from the foreground mask; worse, when they leave, the spot they occupied now differs from the (contaminated) background and flickers as a false positive. Updating only the background-labeled pixels keeps the model honest. Even so, a single mean per pixel cannot represent a background that legitimately takes more than one value, and most real backgrounds do.

2. The Gaussian Mixture Model: MOG2 Intermediate

The dominant classical answer, and still the default in OpenCV, is to model each pixel's history as a mixture of Gaussians (Stauffer and Grimson, 1999; refined by Zivkovic, 2004, into the MOG2 used today). Instead of one mean, each pixel keeps $K$ Gaussian components (typically three to five), each with a weight $w_k$, mean $\mu_k$, and variance $\sigma_k^2$. The recurring background values, leaf and sky, water and spray, occupy the high-weight, low-variance components; rare or transient values do not. A new pixel value matches a component if it lands within a few standard deviations of that component's mean. The components are sorted by $w_k / \sigma_k$ (high weight and low spread first), and the top components whose weights sum past a fraction $T$ of the total are declared "background". A pixel is foreground when it matches none of the background components.

On each frame, a matched component nudges its mean and variance toward the new value and increases its weight, while unmatched components decay; if nothing matches, the weakest component is replaced by a new Gaussian centered on the current value. Written out, with learning rate $\alpha$ and a match indicator $M_k$ (one for the matched component, zero otherwise), the per-pixel online update for the new value $x_t$ is

where $\rho \approx \alpha / w_k$ is the per-component rate (faster for low-weight components, so a new mode settles quickly). Every weight decays by $(1-\alpha)$ each frame and only the matched component is pushed back up by $\alpha$, so a value that recurs accumulates weight while a one-off (a person crossing) never does; the weights are renormalized to sum to one after the step. This online update is why a swaying branch's two states both survive as background components, while a person walking through, an unrepeated value, never accumulates enough weight to be accepted. Figure 15.4.1 contrasts the unimodal and mixture views of a single oscillating pixel.

OpenCV ships two production background subtractors, MOG2 (the Gaussian mixture above) and KNN (a non-parametric k-nearest-neighbor model that compares each pixel against a buffer of recent samples). Both expose a detectShadows option and an online learning rate. Code Fragment 2 runs MOG2 and then cleans its raw mask with the morphological opening and closing of Chapter 6, which is essential because raw subtraction masks are always speckled with noise pixels and pocked with holes.

import cv2

cap = cv2.VideoCapture("street.mp4")

# history: frames of memory; varThreshold: Mahalanobis^2 match gate;
# detectShadows: label soft shadows as gray (127) instead of foreground
mog2 = cv2.createBackgroundSubtractorMOG2(
    history=500, varThreshold=16, detectShadows=True)

kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5))

while True:
    ok, frame = cap.read()
    if not ok:
        break
    fg = mog2.apply(frame, learningRate=-1)   # -1 = auto rate from history

    fg[fg == 127] = 0                          # drop shadow pixels (kept as 127)
    fg = cv2.morphologyEx(fg, cv2.MORPH_OPEN, kernel)   # remove speckle
    fg = cv2.morphologyEx(fg, cv2.MORPH_CLOSE, kernel)  # fill small holes

    # connected components give per-object blobs ready for tracking (Section 15.5)
    n, labels, stats, centroids = cv2.connectedComponentsWithStats(fg)
    movers = [i for i in range(1, n) if stats[i, cv2.CC_STAT_AREA] > 400]
    print(f"frame: {len(movers)} moving blobs above 400 px")
# Typical street clip, one printed line per frame:
# frame: 0 moving blobs above 400 px   (empty street)
# frame: 3 moving blobs above 400 px   (two pedestrians and a car)

3. Shadows, Light Changes, and the Chromaticity Trick Intermediate

The most persistent practical failure of background subtraction is the cast shadow. A person's shadow on the floor differs from the remembered background, so a naive detector labels the shadow as foreground and the silhouette balloons, merging neighboring people and ruining any downstream counting or tracking. The fix exploits a physical fact: a shadow scales a surface's brightness down roughly uniformly across color channels while leaving its chromaticity (the ratio of channels, the surface's color) nearly unchanged. A true foreground object, by contrast, usually changes the color, not just the brightness. So a candidate foreground pixel is reclassified as shadow when its value is a darkened version of the background value at the same hue and saturation, which is exactly the test MOG2's detectShadows applies in a normalized color space. We dropped those shadow-labeled pixels (value 127) in Code Fragment 2.

The harder failure is the sudden global light change: someone flips a light switch, or the sun emerges, and every pixel changes at once. No incremental model can adapt within a single frame, so the entire image flares as foreground for the seconds it takes the learning rate to catch up. Robust systems detect this case explicitly (when more than, say, 70 percent of pixels are flagged foreground at once, treat it as a global illumination event, not an army of intruders) and respond by temporarily boosting the learning rate to re-seed the background fast. This connects back to the photometric normalization ideas of Chapter 2: working in a normalized or gradient-based representation, rather than raw intensity, buys robustness to the lighting changes that wreck raw-intensity models.

4. When the Camera Moves: Compensated and Panoramic Backgrounds Advanced

Every model so far assumed a fixed camera. The instant the camera pans, tilts, or rides on a moving platform, the static-background assumption collapses: the whole scene shifts in the image, so every pixel differs from its remembered value and the entire frame reads as foreground. There are two classical escapes. The first is motion compensation: estimate the dominant frame-to-frame motion as a global transform, a homography from Chapter 5 for a rotating or distant-scene camera, warp the previous background into the current frame's coordinates, and only then subtract. What remains after compensating the camera's own motion is independent motion, objects moving differently from the static world, which is exactly what you wanted to detect. The global transform is estimated with the same RANSAC-on-keypoint-matches machinery from Chapter 10, or from the sparse optical flow of Section 15.2 by fitting a model to the dominant flow vectors.

The second escape, for a camera that pans across a fixed scene (a security pan-tilt-zoom (PTZ) unit on a tour, for instance), is to build a panoramic background: stitch the views into one large mosaic, register each incoming frame into the mosaic via homography, and subtract against the corresponding mosaic region. This turns a moving-camera problem back into a static-background problem in mosaic coordinates. Code Fragment 3 sketches the compensation idea: estimate the inter-frame homography, warp the running background forward, and subtract.

import cv2
import numpy as np

prev_gray = None
bg = None
orb = cv2.ORB_create(1000)
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)

def estimate_homography(g0, g1):
    """Dominant global motion g0 -> g1 as a homography (Chapter 5 / Chapter 10)."""
    k0, d0 = orb.detectAndCompute(g0, None)
    k1, d1 = orb.detectAndCompute(g1, None)
    m = sorted(bf.match(d0, d1), key=lambda x: x.distance)[:300]
    p0 = np.float32([k0[x.queryIdx].pt for x in m])
    p1 = np.float32([k1[x.trainIdx].pt for x in m])
    H, _ = cv2.findHomography(p0, p1, cv2.RANSAC, 3.0)   # robust to movers
    return H

cap = cv2.VideoCapture("panning_camera.mp4")
while True:
    ok, frame = cap.read()
    if not ok:
        break
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    if prev_gray is None:
        bg = gray.astype(np.float32)
    else:
        H = estimate_homography(prev_gray, gray)         # camera motion
        h, w = gray.shape
        bg = cv2.warpPerspective(bg, H, (w, h))          # align bg to new frame
        diff = cv2.absdiff(gray.astype(np.float32), bg)
        fg = (diff > 25).astype(np.uint8) * 255          # residual = movers
        bg = 0.98 * bg + 0.02 * gray                     # adapt in new frame
        cv2.imshow("independent motion", fg)
        if cv2.waitKey(1) == 27:
            break
    prev_gray = gray

The catch in Code Fragment 3 is that a single homography is exact only when the scene is planar or the camera rotates about its optical center. For a freely translating camera viewing a 3D scene, parallax makes the global model leak, near objects move more than far ones, so background near depth discontinuities is misclassified as motion. Handling that properly requires the multi-view geometry of Chapter 13 and Chapter 14, or it is simply deferred to the learned motion-segmentation models that the research frontier below describes.