Motion, Optical Flow & Tracking

Chapter Overview

Every chapter so far has treated the image as a frozen instant. Chapter 14 came closest to motion: it moved a camera through a static world and recovered geometry from the parallax. This chapter inverts the arrangement. Now the world itself moves, and the question is no longer "where is the camera?" but "what is moving, where is it going, and which thing is it?" Those are genuinely different problems. A pixel that brightens might be an object arriving, a shadow leaving, or a cloud uncovering the sun, and nothing in a single frame can tell you which. Time is a new measurement axis, and like every measurement axis it comes with its own noise, its own aliasing, and its own beautiful, treacherous ambiguities.

The first half of the chapter is about pixel motion. Section 15.1 lays the foundations: the distinction between the motion field (the geometric truth) and optical flow (what brightness patterns appear to do), the brightness constancy assumption that links them, and the aperture problem that makes flow locally unknowable in one direction. Section 15.2 turns one unsolvable equation into a solvable system by assuming flow is constant over a small window: the Lucas-Kanade method, whose normal-equations matrix turns out to be exactly the structure tensor of Chapter 10. Corners, it emerges, are not just matchable; they are trackable, and the KLT tracker built on this insight has run for four decades. Section 15.3 goes dense: Horn-Schunck's global energy, the smoothness prior that fills in flow where data is silent, and the variational lineage that leads to the modern OpenCV workhorses.

The second half is about object motion. Section 15.4 exploits the easiest special case, a static camera, where modeling each pixel's history separates the boring background from the interesting foreground: frame differencing, running Gaussians, and the mixture-of-Gaussians models that survive waving trees and flickering lamps. Section 15.5 follows a single chosen object through the frame: mean-shift climbing a color-histogram likelihood, correlation filters matching templates at Fourier speed, and the drift-versus-adaptation dilemma that haunts every template update. Section 15.6 adds the missing ingredients for multi-object tracking: the Kalman filter, which predicts where each object will be and how sure to be about it, and data association, which decides via the Hungarian algorithm which detection belongs to which track. The SORT family assembled from these pieces still underpins production trackers today.

One theme runs through all six sections. Motion estimation is ill-posed: the data never fully determines the answer, so every method is an assumption about constancy plus a prior about smoothness, wrapped around the same least-squares core. Lucas-Kanade assumes local constancy; Horn-Schunck assumes global smoothness; background subtraction assumes temporal stationarity; the Kalman filter assumes dynamic linearity. When the assumption holds, the method works; when it breaks, the failure is diagnosable. Learning that diagnosis is the real skill this chapter teaches, and it transfers intact to the deep methods of Chapter 26, which replace the hand-chosen priors with learned ones but inherit every one of the underlying ambiguities.