OpenCV Beyond the Basics: features2d, calib3d & video

"People think I am one library. I am forty modules in a trench coat, and the one with the patent-free descriptors moved out years ago. Check the door label before you knock."

An OpenCV Build, Counting Its Modules

Throughout Part II we reached for OpenCV one function at a time: cv2.SIFT_create in Chapter 10, cv2.calibrateCamera in Chapter 12, cv2.findEssentialMat in Chapter 13, cv2.calcOpticalFlowPyrLK in Chapter 15. This section steps back and surveys the three modules those calls came from as coherent toolkits, so that the next time a geometric task appears you can guess, correctly, that OpenCV already solved it. We begin with the packaging decision that gates everything else (the illustration below sketches the three-layer workshop this section surveys).

1. The Packaging Trap: Four Wheels, One Import Name Beginner

Before any function matters, the right OpenCV must be installed, and this is where most newcomers lose an hour. The project ships four mutually exclusive PyPI wheels, all imported as cv2, and installing the wrong one makes perfectly correct code raise an AttributeError. The split exists because some classic algorithms (notably SIFT's relatives and the extra trackers) live in a separate "contrib" repository, and because headless servers want a build without GUI dependencies. The illustration below captures the trap: four wheels knock on the same door, but only the one you installed answers with the function you need.

Since the relevant patents expired, SIFT moved from contrib back into the main module, so cv2.SIFT_create() works in every wheel; SURF (cv2.xfeatures2d.SURF_create) and the newer legacy trackers still require a contrib build. Code 17.1.1 is the first thing to run in any new environment: it prints the build, the version, and a quick capability probe.

# Environment probe: confirm the OpenCV version, whether this is a
# contrib build, that SIFT and ORB are importable, and whether the
# optimized SIMD/OpenCL fast paths are compiled in.
import cv2

print("OpenCV version:", cv2.__version__)
# Is this a contrib build? Probe a contrib-only symbol.
has_contrib = hasattr(cv2, "xfeatures2d")
print("contrib modules present:", has_contrib)

# These are in the MAIN module since the SIFT patent expired in 2020:
sift = cv2.SIFT_create()
orb = cv2.ORB_create()
print("SIFT and ORB available:", sift is not None and orb is not None)

# Optimized backend status (IPP / OpenCL); affects Section 17.1 timings.
print("optimized build:", cv2.useOptimized())

OpenCV version: 4.10.0
contrib modules present: True
SIFT and ORB available: True
optimized build: True

2. features2d: The Detector, Descriptor & Matcher Zoo Intermediate

The features2d module formalizes the pipeline of Chapter 10 into three plug-compatible stages: a detector finds keypoints, a descriptor turns each keypoint's neighborhood into a vector, and a matcher pairs descriptors across images. The elegance is that every detector exposes the same detectAndCompute interface, so swapping SIFT for ORB is a one-line change. Table 17.1.2 inventories the production-relevant choices and the matcher each one demands.

The matcher pairs with the descriptor type: float descriptors use the L2 norm, binary descriptors use the Hamming distance (a bit count, computed in hardware on modern CPUs). Getting this wrong is a frequent bug, an L2 matcher on binary descriptors compiles, runs, and produces nonsense matches. Code 17.1.2 shows the canonical robust matching recipe: detect, describe, match with a k-nearest-neighbor query, then apply Lowe's ratio test to keep only confident matches, exactly the procedure derived in Chapter 10. The illustration below makes the mismatch concrete: a round float plug forced into a square binary socket runs anyway, and quietly returns garbage.

# Robust two-image matching: detect and describe keypoints with SIFT,
# pair them with a k-nearest-neighbour brute-force matcher, then keep
# only the matches that survive Lowe's ratio test.
import cv2

img1 = cv2.imread("view1.jpg", cv2.IMREAD_GRAYSCALE)
img2 = cv2.imread("view2.jpg", cv2.IMREAD_GRAYSCALE)

# Detector + descriptor in one object; one line to swap algorithms.
detector = cv2.SIFT_create(nfeatures=4000)
kp1, des1 = detector.detectAndCompute(img1, None)
kp2, des2 = detector.detectAndCompute(img2, None)

# SIFT is float -> L2 norm. For ORB/AKAZE/BRISK use cv2.NORM_HAMMING.
bf = cv2.BFMatcher(cv2.NORM_L2)
knn = bf.knnMatch(des1, des2, k=2)            # two nearest neighbours each

# Lowe's ratio test: keep a match only if the best is clearly better than 2nd.
good = [m for m, n in knn if m.distance < 0.75 * n.distance]
print(f"{len(kp1)} + {len(kp2)} keypoints -> {len(good)} confident matches")

3812 + 3955 keypoints -> 642 confident matches

For large image collections the brute-force matcher becomes the bottleneck. OpenCV ships cv2.FlannBasedMatcher, an approximate nearest-neighbor index that trades a little recall for a large speedup, the same engine that scales the matching stage inside the reconstruction tools of Section 17.2.

3. calib3d: The Geometry Verbs Advanced

If features2d produces correspondences, calib3d turns them into geometry. This is the densest module in Part II's story: it implements camera calibration (Chapter 12), the fundamental and essential matrices and pose recovery (Chapter 13), the Perspective-n-Point (PnP) problem (recovering a single camera's pose from known 3D points and their 2D projections), triangulation, and dense stereo. Figure 17.1.1 organizes the module by the geometric question each function answers, which is the mental index worth memorizing.

The arrows in Figure 17.1.1 are also the order to memorize the module, a five-question chain that mirrors how a reconstruction actually proceeds: what lens, how two views relate, where the camera is, where the point is, how far every pixel. Each answer feeds the next (calibration yields the intrinsics two-view estimation needs, which yields the poses triangulation and stereo consume), so recalling the order recalls the data dependencies for free.

A recurring detail trips up everyone once: most calib3d functions return a rotation as a 3-element Rodrigues vector (an axis-angle encoding), not a $3 \times 3$ matrix. The conversion is $R = $ cv2.Rodrigues(rvec)[0], and a rotation $R$ with translation $\mathbf{t}$ composes the projection $\mathbf{x} \sim K [R \mid \mathbf{t}] \mathbf{X}$ from Chapter 12. Code 17.1.3 chains the verbs into a minimal two-view pose recovery, the computational core of the structure-from-motion of Chapter 14.

# Two-view pose recovery: estimate the essential matrix from matched
# points, decompose it into relative rotation and translation, then
# triangulate the inlier correspondences into 3D coordinates.
import cv2
import numpy as np

# pts1, pts2: matched, undistorted image points (Nx2 float32); K: intrinsics.
E, mask = cv2.findEssentialMat(pts1, pts2, K,
                               method=cv2.USAC_MAGSAC,   # modern robust estimator
                               prob=0.999, threshold=1.0)

# Decompose E into relative pose, keeping only the cheirality-valid solution.
n_inliers, R, t, mask_pose = cv2.recoverPose(E, pts1, pts2, K, mask=mask)
print(f"recovered pose from {n_inliers} inliers")

# Triangulate the inliers into 3D (homogeneous), given the two camera matrices.
P1 = K @ np.hstack([np.eye(3), np.zeros((3, 1))])     # first camera at origin
P2 = K @ np.hstack([R, t])                            # second camera pose
inl = mask_pose.ravel().astype(bool)
X_h = cv2.triangulatePoints(P1, P2, pts1[inl].T, pts2[inl].T)
X = (X_h[:3] / X_h[3]).T                              # de-homogenize -> Nx3
print("triangulated points:", X.shape)

recovered pose from 588 inliers
triangulated points: (588, 3)

3.1 Dense Stereo in One Object

For rectified stereo pairs, cv2.StereoSGBM_create implements the semi-global matching of Chapter 13: it computes a per-pixel disparity by aggregating matching costs along multiple 1D paths, an efficient approximation to a full 2D smoothness optimization. The disparity $d$ converts to depth $Z$ by $Z = f B / d$, where $f$ is the focal length and $B$ the baseline (the distance between the two cameras); the inverse relationship is the intuition that near objects shift a lot between the two views while far ones barely move, exactly as a nearby finger jumps more than the horizon when you blink one eye at a time. Figure 17.1.2 makes that inverse law geometric: two rectified cameras view the same point, and the closer the point, the wider the angle between the two sightlines and the larger the disparity it produces on the image planes. The block size and the two smoothness penalties $P_1, P_2$ are the parameters worth tuning; the defaults rarely suit a new camera.

4. video: Flow, Trackers & the Kalman Filter Intermediate

Where features2d and calib3d reason about geometry inside a single frame or across a static pair, the last module turns to what changes between frames. The video module is the temporal half of OpenCV and the home of Chapter 15. It splits into three families: sparse and dense optical flow, object trackers, and the cv2.KalmanFilter state estimator. Sparse flow (cv2.calcOpticalFlowPyrLK) follows a handful of corners across frames using the pyramidal Lucas-Kanade method; dense flow (cv2.calcOpticalFlowFarneback, or the contrib cv2.optflow algorithms) estimates a motion vector at every pixel. Code 17.1.4 is the standard sparse-flow loop: find good corners to track, then propagate them with a forward-backward consistency check that discards points the tracker lost.

# Sparse optical-flow loop: seed Shi-Tomasi corners on the first frame,
# then track them frame to frame with pyramidal Lucas-Kanade, keeping
# only the points the tracker reports as successfully followed.
import cv2
import numpy as np

cap = cv2.VideoCapture("walk.mp4")
ok, prev = cap.read()
prev_gray = cv2.cvtColor(prev, cv2.COLOR_BGR2GRAY)
# Shi-Tomasi corners are the classic features to track (Chapter 15).
p0 = cv2.goodFeaturesToTrack(prev_gray, maxCorners=200, qualityLevel=0.3,
                             minDistance=7)

while True:
    ok, frame = cap.read()
    if not ok:
        break
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    # Pyramidal Lucas-Kanade: track p0 forward; st==1 marks successful tracks.
    p1, st, err = cv2.calcOpticalFlowPyrLK(prev_gray, gray, p0, None)
    good_new = p1[st.ravel() == 1]
    # ... draw or store the tracks here ...
    prev_gray, p0 = gray, good_new.reshape(-1, 1, 2)
cap.release()

The tracker family wraps higher-level single-object trackers behind a uniform init/update interface. The correlation-filter tracker cv2.TrackerKCF_create and the more accurate cv2.TrackerCSRT_create live in the main wheel; several others moved to cv2.legacy in OpenCV 4.5. The cv2.KalmanFilter class implements the constant-velocity and constant-acceleration state estimators of Chapter 15, the predict-correct recursion that smooths a noisy detector into a stable track and bridges short occlusions.

That last property, bridging an occlusion, is worth unpacking, because it is not a special case but a direct consequence of the two-step structure. When the detector returns nothing for a frame, the filter simply runs the predict step (advance the state by the motion model) and skips the correct step, so it coasts on its last estimated velocity and keeps emitting a plausible position until a detection reappears and pulls the estimate back. That same predict-without-correct mechanism is why the filter degrades gracefully under a few missed frames instead of dropping the track entirely.