Image Registration & Alignment

"Four hundred of us pointed at the wrong building, and the algorithm still found the homography. I demand a recount."

A Feature Point That Refused to Match

So far in this chapter, someone always handed us the transform: we chose rotation angles, picked corner points, invented distortion coefficients. Production systems rarely enjoy that luxury. A panorama stitcher receives two overlapping photos and must discover the homography between them; a video stabilizer must discover the inter-frame motion; a remote-sensing pipeline must align today's satellite pass with last month's. This estimation problem is image registration, one of the oldest and most economically consequential problems in vision, and it ties the entire chapter together: the hierarchy of Section 5.1 tells us what to estimate, and the warping machinery of Section 5.4 executes whatever we find.

1. Phase Correlation: Alignment From the Fourier Domain Intermediate

For pure translation there is a method of almost unreasonable elegance. The Fourier shift theorem from Chapter 4 says that shifting an image by $(\Delta x, \Delta y)$ multiplies its Fourier transform by a phase ramp $e^{-i2\pi(u\Delta x + v\Delta y)}$ while leaving magnitudes untouched. So if $F_1$ and $F_2$ are the transforms of two shifted copies, the normalized cross-power spectrum isolates exactly that ramp:

The inverse transform of $R$ is, ideally, a single bright spike sitting at the shift. Find the spike, read off the translation; subpixel accuracy comes from fitting the peak's neighborhood. Because the normalization throws away all magnitude information, the method is remarkably immune to global illumination changes, and it considers every pixel at once rather than trusting any local landmark:

import cv2
import numpy as np

f1 = np.float32(cv2.imread("pass_may.png",  cv2.IMREAD_GRAYSCALE))
f2 = np.float32(cv2.imread("pass_june.png", cv2.IMREAD_GRAYSCALE))

# Hanning window suppresses the spectral leakage from image borders
win = cv2.createHanningWindow(f1.shape[::-1], cv2.CV_32F)
(dx, dy), response = cv2.phaseCorrelate(f1, f2, win)
print(f"shift = ({dx:+.2f}, {dy:+.2f}) px,  peak response = {response:.3f}")

shift = (+17.38, -4.92) px,  peak response = 0.871

The translation-only limitation has a classical workaround worth knowing: resample the Fourier magnitude spectra to log-polar coordinates, where rotation and scaling become translations, run phase correlation there to recover them, undo them, then run phase correlation again for the shift. This Fourier-Mellin pipeline (OpenCV provides cv2.warpPolar for the resampling) handles the full similarity group of Section 5.1 and is a staple of scan-matching and watermark-detection systems.

2. Direct Methods: ECC Alignment Intermediate

Phase correlation is global and rigid; the next family treats registration as continuous optimization. Direct (intensity-based) methods parameterize a warp $W_\mathbf{p}$ by the entries of a transformation matrix and search for the parameters that make the warped moving image best match the fixed image, descending the gradient of a similarity objective. The standard-bearer in OpenCV is ECC (Evangelidis and Psarakis, 2008), which maximizes the enhanced correlation coefficient, a similarity measure invariant to global brightness gain and bias, so a darker exposure of the same scene still registers cleanly:

fixed  = cv2.imread("band_nir.png",  cv2.IMREAD_GRAYSCALE)
moving = cv2.imread("band_red.png",  cv2.IMREAD_GRAYSCALE)

warp = np.eye(2, 3, dtype=np.float32)              # identity initialization
criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 200, 1e-6)

cc, warp = cv2.findTransformECC(fixed, moving, warp,
                                motionType=cv2.MOTION_EUCLIDEAN,
                                criteria=criteria)
print(f"final correlation: {cc:.4f}")
aligned = cv2.warpAffine(moving, warp, fixed.shape[::-1],
                         flags=cv2.INTER_LINEAR + cv2.WARP_INVERSE_MAP)

Direct methods have a sharply defined personality. They use every pixel's gradient, so they achieve excellent subpixel accuracy and work on images with no distinctive corners at all (smooth terrain, medical slices, blurry frames). But the optimization is local: start it more than a few dozen pixels from the truth and it converges to the wrong valley. The standard remedy is coarse-to-fine: run ECC on the smallest level of a Gaussian pyramid from Chapter 4, where the basin of convergence is wide, and propagate the result down the levels as initialization. Phase correlation also makes an excellent initializer. When we reach optical flow in Chapter 15, you will recognize Lucas-Kanade as exactly this machinery applied to small patches.

3. Feature-Based Registration and RANSAC Intermediate

The third family is the one inside every panorama app. Instead of comparing all pixels, find a few hundred distinctive points in each image, describe each by a compact signature of its neighborhood, match signatures across images, and fit a transform to the matched pairs. The detection and description machinery (ORB here; SIFT, AKAZE, and learned successors) gets a full treatment in Chapter 10; today we use it as a black box that outputs correspondences, because the interesting part for this chapter is what happens next.

What happens next is that many correspondences are wrong. Repetitive windows match the wrong window; foliage matches other foliage; 10 to 70 percent outlier rates are routine. Least-squares fitting averages outliers into the answer and is destroyed by them. The fix, RANSAC (Random Sample Consensus, Fischler and Bolles, 1981), is one of the most-used algorithms in all of computer vision:

Input: correspondences C, model family with minimal sample size s,
       inlier threshold t (px), iterations N
Best := empty
Repeat N times:
    1. Draw s correspondences from C at random
       (s = 1 translation, 2 similarity, 3 affine, 4 homography)
    2. Fit the model exactly to those s pairs
    3. Count how many correspondences in C the model maps
       within t pixels of their partner: the inlier set
    4. If this inlier set is the largest so far, keep it
Refit the model by least squares on the final inlier set
Return model + inlier mask

The logic is probabilistic: a random minimal sample is entirely outlier-free with probability $w^s$ (where $w$ is the inlier fraction), and one clean sample is enough, because the model it generates will be confirmed by all the other inliers. To find at least one clean sample with confidence $p$, you need

iterations: for $w = 0.5$, $s = 4$, $p = 0.99$, about 72 draws, microseconds of work. Figure 5.5.1 shows the geometry of why counting inliers separates the true model from outlier-contaminated fits.

OpenCV bundles the entire loop, including the final least-squares polish, inside cv2.findHomography. A complete feature-based registration pipeline is then short enough to memorize:

img1 = cv2.imread("left.jpg",  cv2.IMREAD_GRAYSCALE)   # moving
img2 = cv2.imread("right.jpg", cv2.IMREAD_GRAYSCALE)   # fixed

orb = cv2.ORB_create(nfeatures=4000)
k1, d1 = orb.detectAndCompute(img1, None)
k2, d2 = orb.detectAndCompute(img2, None)

bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
matches = sorted(bf.match(d1, d2), key=lambda m: m.distance)[:600]

src = np.float32([k1[m.queryIdx].pt for m in matches]).reshape(-1, 1, 2)
dst = np.float32([k2[m.trainIdx].pt for m in matches]).reshape(-1, 1, 2)

H, inlier_mask = cv2.findHomography(src, dst, cv2.RANSAC,
                                    ransacReprojThreshold=3.0)
n_in = int(inlier_mask.sum())
print(f"RANSAC kept {n_in}/{len(matches)} matches as inliers")

aligned = cv2.warpPerspective(img1, H, img2.shape[::-1])

RANSAC kept 311/600 matches as inliers

4. Choosing a Method, Choosing a Model Beginner

The three families are complements, not competitors. Table 5.5.1 is the decision matrix this book will reuse whenever alignment appears.

Orthogonal to method choice is model choice, and here Section 5.1's discipline pays its dividend: fit the smallest family the physics permits. A microscope stage needs translation; a scanning satellite needs similarity or affine; overlapping handheld photos of a facade need a homography. And when the scene has genuine depth parallax between camera positions, no global 2D model is correct: the best homography will align the dominant plane and ghost everything off it, which is the cue to graduate to the epipolar geometry of Chapter 13 or to local deformable models.

stitcher = cv2.Stitcher_create(cv2.Stitcher_PANORAMA)
status, pano = stitcher.stitch([img_a, img_b, img_c])
if status == cv2.Stitcher_OK: cv2.imwrite("pano.jpg", pano)

Registration completes the chapter's toolkit: we can now describe transforms, compose them, execute them, and recover them from data. The closing section spends the entire budget at once. We will build a document scanner from scratch, and every stage of it, model selection, corner correspondence, homography estimation, inverse warping, interpolation, will be a tool you can now name. Continue to Section 5.6.