Calibration Workflows, Targets & Quality Checks

"My calibration was perfect in June. Since then I have ridden in three delivery trucks, baked on a dashboard, and someone 'just adjusted' my focus ring. I no longer know who I am."

A Slightly Decalibrated Production Camera

The illustration above is the failure this whole section is built to prevent: a once-crisp camera worn out of calibration by the months of vibration, temperature, and well-meaning colleagues that follow installation.

Everything so far in this chapter has assumed the calibration data was good. In practice, data quality is where calibrations are won and lost: the same solver, fed a lazy capture session, returns parameters that are confidently wrong in ways that surface weeks later as bent point clouds in Chapter 13 or drifting maps in Chapter 14. This section is the field manual: targets, capture protocol, diagnostics, and operations.

1. Choosing a Target Basic

Three families of planar targets dominate practice, compared in Figure 12.5.1. The classic checkerboard offers the best localization primitive in vision: a saddle point between four squares, locatable to a few hundredths of a pixel and unbiased under perspective and modest distortion. Its weakness is bookkeeping: the standard detector requires the entire board visible in every frame, which fights directly against the advice to push the board into the image corners. The circle grid detects fast via blob centroids, but a circle images as an ellipse whose centroid is not the projected circle center; under strong perspective or distortion this bias contaminates exactly the measurements calibration cares about. The modern default is the ChArUco board: a checkerboard whose white squares carry small ArUco markers (from Section 12.4), so every visible corner is individually identified. Partial views work, occlusion works, and corners keep checkerboard-grade subpixel accuracy.

The physical print matters as much as the pattern. Paper taped to a wall ripples; a 1 mm bow on a 30 cm board is a 1-part-in-300 violation of the planarity assumption at the heart of Zhang's math, comfortably enough to corrupt the third decimal of your focal length. Mount targets on glass or aluminum composite panel, verify the printed square size with calipers (printers rescale silently, as noted in Section 12.3), and prefer matte lamination to glossy, which blooms under lights.

2. A Capture Protocol That Constrains Every Parameter Intermediate

Each parameter of the camera model is constrained by a specific kind of view, so a capture session is not "take 20 pictures", it is "buy information for every parameter." The working checklist:

• 15 to 30 accepted views. Below 10, the LM optimum is noise-shaped; beyond 30, returns diminish unless coverage is still improving.
• Tilt the board, up to about 45 degrees, in both axes. Tilted views are what separate focal length from distance (the degeneracy from Section 12.3); a session of fronto-parallel views leaves $f_x, f_y$ nearly unobservable, however many images you take.
• Cover the full frame, especially corners. Distortion coefficients are fitted where corners are observed; a center-only dataset extrapolates the polynomial into the corners, and extrapolated polynomials do what extrapolated polynomials always do. With ChArUco, deliberately hang the board partially out of frame.
• Vary the distance over at least a factor of two around the working distance of your application.
• Lock focus, lock exposure, kill autofocus and stabilization. Autofocus changes the focal length between frames (focus breathing); optical stabilization moves the lens, shifting the principal point per frame. Both make "the" intrinsics a moving target.
• Reject blur. Motion blur shifts detected corners by more than the subpixel accuracy you are paying for. Hold the board still, or light the scene to allow short exposures; a variance-of-Laplacian sharpness gate from Chapter 1's toolbox makes rejection automatic.

3. ChArUco Calibration with the Modern API Intermediate

OpenCV 4.7 reorganized the ArUco interfaces around detector objects (the same release that introduced the ArucoDetector of Section 12.4), and ChArUco calibration became pleasantly short: a CharucoDetector finds and identifies corners, and the board object itself converts them into matched 3D-2D arrays for the standard calibrateCamera. No manual corner-ID bookkeeping survives in user code.

# ChArUco calibration with the OpenCV 4.7+ object API: CharucoDetector finds
# identified corners and board.matchImagePoints turns them into 3D-2D arrays.
# Because corners are individually identified, partial board views still count.
import glob
import cv2
import numpy as np

aruco = cv2.aruco
dictionary = aruco.getPredefinedDictionary(aruco.DICT_5X5_100)
# 7 x 5 squares, 35 mm squares, 26 mm markers: measure your actual print.
board = aruco.CharucoBoard((7, 5), 0.035, 0.026, dictionary)
detector = aruco.CharucoDetector(board)

all_obj, all_img, used = [], [], 0
for path in sorted(glob.glob("charuco/*.png")):
    gray = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
    ch_corners, ch_ids, _, _ = detector.detectBoard(gray)
    if ch_ids is None or len(ch_ids) < 8:        # too few corners constrains nothing
        continue
    obj_pts, img_pts = board.matchImagePoints(ch_corners, ch_ids)
    all_obj.append(obj_pts)
    all_img.append(img_pts)
    used += 1

rms, K, dist, rvecs, tvecs = cv2.calibrateCamera(
    all_obj, all_img, gray.shape[::-1], None, None)
print(f"{used} usable views, RMS = {rms:.3f} px")
print("fx, fy =", round(K[0, 0], 1), round(K[1, 1], 1))
# 26 usable views, RMS = 0.164 px
# fx, fy = 1539.1 1538.6

4. Quality Beyond RMS: Residual Forensics Intermediate

The RMS is a single number summarizing thousands of residual vectors, and the vectors know more than their average. Three diagnostics, in increasing order of revelation. First, per-view errors (built in Section 12.3) expose individual bad frames. Second, parameter standard deviations from calibrateCameraExtended expose unconstrained parameters. Third, and most informative, the residual quiver plot: draw every corner's residual as an arrow at its image position. A healthy calibration shows structureless arrows, isotropic noise with no preferred direction. Structure is a confession: arrows swirling radially near the corners mean the distortion model has too few terms (the model-family failure from Section 12.2); arrows that flip direction across the board in one view mean that board moved or its print is not flat; a global drift pattern points at rolling shutter or synchronization issues.

# Build the residual quiver plot: collect every corner's reprojection residual
# vector and draw it as an arrow at its image position. Structureless arrows
# mean a good fit; radial swirls or per-view drift name a specific failure.
import matplotlib.pyplot as plt

pos, res = [], []
for op, ip, rv, tv in zip(all_obj, all_img, rvecs, tvecs):
    proj, _ = cv2.projectPoints(op, rv, tv, K, dist)
    pos.append(ip.reshape(-1, 2))
    res.append((ip - proj).reshape(-1, 2))
pos, res = np.vstack(pos), np.vstack(res)

plt.figure(figsize=(9, 5.5))
# scale=0.02 in xy units draws arrows 50x their true (subpixel) length.
plt.quiver(pos[:, 0], pos[:, 1], res[:, 0], res[:, 1],
           angles="xy", scale_units="xy", scale=0.02, width=0.002)
plt.gca().invert_yaxis()                      # image coordinates: y grows down
plt.title("Reprojection residuals (arrows exaggerated 50x)")
plt.savefig("residual_quiver.png", dpi=150)

print(f"mean |residual| = {np.linalg.norm(res, axis=1).mean():.3f} px")
print(f"99th percentile = {np.percentile(np.linalg.norm(res, axis=1), 99):.3f} px")
# mean |residual| = 0.139 px
# 99th percentile = 0.501 px

A final validation habit borrowed from machine learning: hold out three or four views, calibrate without them, and reproject onto the held-out corners. Held-out error close to training error means the calibration generalizes; held-out error several times larger means the model memorized a too-small or too-uniform capture set, which is overfitting in its photogrammetric costume, the same generalization logic that recurs throughout Chapter 21.

5. Calibration as an Operational Asset Basic

A calibration that lives in a variable inside one script dies with that script. Persist it with metadata sufficient to audit it later: the matrix and distortion vector, image size (intrinsics are resolution-specific), RMS and per-view statistics, capture date, and the identity of the physical camera and lens, because, as the gauging-station story in Section 12.3 showed, calibration binds to a serial number, not to a product model. OpenCV's FileStorage writes YAML or XML that every OpenCV binding reads back natively.

# Persist a calibration with the metadata needed to audit it later: the matrix
# and distortion, the resolution they are valid for, RMS, and the physical
# camera serial and date, then read it back to confirm a lossless round trip.
fs = cv2.FileStorage("cam_LX4407_2026-06-11.yml", cv2.FILE_STORAGE_WRITE)
fs.write("image_width", 1920); fs.write("image_height", 1080)
fs.write("K", K); fs.write("dist", dist)
fs.write("rms_px", rms)
fs.write("camera_serial", "LX-4407"); fs.write("lens", "8mm-f1.8, focus locked")
fs.write("calibrated_on", "2026-06-11")
fs.release()

fs = cv2.FileStorage("cam_LX4407_2026-06-11.yml", cv2.FILE_STORAGE_READ)
K_loaded = fs.getNode("K").mat()
dist_loaded = fs.getNode("dist").mat()
fs.release()
assert np.allclose(K_loaded, K)
print("calibration restored:", K_loaded[0, 0], "px focal")
# calibration restored: 1539.1 px focal

Finally, know the recalibration triggers by heart: any lens or focus change, any transport or impact, any mount adjustment, large temperature excursions, and (for intrinsics) any change of streaming resolution or crop mode. Rolling-shutter cameras deserve a special caution: if the board (or camera) moves during the row-by-row exposure, corners within one frame are sampled at different instants and the board is effectively non-rigid; calibrate rolling-shutter cameras with both the camera and the target stationary per shot. These operational habits compound: teams that version calibrations, gate them on diagnostics, and monitor reprojection in production spend their debugging time on actual vision problems instead of phantom geometry bugs in Chapter 14-style pipelines.

The exercises above sharpen individual instincts; the Hands-On Lab below assembles the chapter's full three-beat arc, project, calibrate, and locate, into a single script you can run with no camera and no downloaded images.

pip install numpy opencv-python

import numpy as np
import cv2

W, H = 1280, 960
K_true = np.array([[1000.0,    0.0, 640.0],
                   [   0.0, 1000.0, 480.0],
                   [   0.0,    0.0,   1.0]])
# TODO: define dist_true as a (5,) array [k1, k2, p1, p2, k3]
# Hint: a mild barrel lens has a negative k1, for example -0.25, with smaller k2, p1, p2, k3
dist_true = ...
print("ground-truth focal:", K_true[0, 0], "px")

COLS, ROWS = 9, 6          # inner corners
SQUARE = 0.025             # 25 mm squares

def board_points(cols, rows, square):
    pts = np.zeros((rows * cols, 3), np.float32)
    # TODO: fill columns 0 and 1 with the x, y grid (column 2 stays 0 for a planar board)
    # Hint: np.mgrid[0:cols, 0:rows].T.reshape(-1, 2) * square
    ...
    return pts

objp = board_points(COLS, ROWS, SQUARE)
print("object points:", objp.shape)   # (54, 3)

rng = np.random.default_rng(0)

def random_pose():
    rvec = rng.uniform(-0.5, 0.5, 3).astype(np.float64)        # tilt in radians
    tvec = np.array([rng.uniform(-0.1, 0.1),
                     rng.uniform(-0.1, 0.1),
                     rng.uniform(0.4, 0.8)], np.float64)        # board 0.4 to 0.8 m away
    return rvec, tvec

image_points = []
for _ in range(20):
    rvec, tvec = random_pose()
    # TODO: project objp with cv2.projectPoints using K_true and dist_true
    # keep only views whose corners all fall inside [0, W) x [0, H)
    ...
print("usable views:", len(image_points))

obj_list = [objp] * len(image_points)
rms, K_est, dist_est, rvecs, tvecs = cv2.calibrateCamera(
    obj_list, image_points, (W, H), None, None)

print(f"RMS reprojection error: {rms:.4f} px")
# TODO: print the focal-length error |K_est[0,0] - K_true[0,0]|
#       and the principal-point error, in pixels
...

noisy_points = [p + rng.normal(0, 0.3, p.shape).astype(np.float32)
                for p in image_points]
# TODO: recalibrate with noisy_points and report the new RMS and focal error
...

rvec_gt, tvec_gt = random_pose()
proj_gt, _ = cv2.projectPoints(objp, rvec_gt, tvec_gt, K_est, dist_est)

# TODO: recover the pose with cv2.solvePnP(objp, proj_gt, K_est, dist_est)
#       then reproject and print the mean pixel reprojection error
...

corrupt = proj_gt.copy()
corrupt[[3, 17, 40]] += rng.uniform(-50, 50, (3, 1, 2)).astype(np.float64)

# A robust pose in one call: RANSAC finds the inliers for you.
ok, rvec_r, tvec_r, inliers = cv2.solvePnPRansac(objp, corrupt, K_est, dist_est)
# TODO: print how many inliers RANSAC kept and confirm the three corrupted
#       indices are absent from the inlier set
...

import numpy as np
import cv2

# ---- Step 1: ground-truth camera ----
W, H = 1280, 960
K_true = np.array([[1000.0,    0.0, 640.0],
                   [   0.0, 1000.0, 480.0],
                   [   0.0,    0.0,   1.0]])
dist_true = np.array([-0.25, 0.10, 0.001, -0.001, 0.02])

# ---- Step 2: board object points ----
COLS, ROWS, SQUARE = 9, 6, 0.025
def board_points(cols, rows, square):
    pts = np.zeros((rows * cols, 3), np.float32)
    pts[:, :2] = np.mgrid[0:cols, 0:rows].T.reshape(-1, 2)
    return pts * square
objp = board_points(COLS, ROWS, SQUARE)

# ---- Step 3: synthesize views ----
rng = np.random.default_rng(0)
def random_pose():
    rvec = rng.uniform(-0.5, 0.5, 3).astype(np.float64)
    tvec = np.array([rng.uniform(-0.1, 0.1), rng.uniform(-0.1, 0.1),
                     rng.uniform(0.4, 0.8)], np.float64)
    return rvec, tvec

image_points = []
while len(image_points) < 20:
    rvec, tvec = random_pose()
    proj, _ = cv2.projectPoints(objp, rvec, tvec, K_true, dist_true)
    p = proj.reshape(-1, 2)
    if (p[:, 0] >= 0).all() and (p[:, 0] < W).all() and \
       (p[:, 1] >= 0).all() and (p[:, 1] < H).all():
        image_points.append(proj.astype(np.float32))

# ---- Step 4: calibrate and compare ----
obj_list = [objp] * len(image_points)
rms, K_est, dist_est, rvecs, tvecs = cv2.calibrateCamera(
    obj_list, image_points, (W, H), None, None)
print(f"RMS reprojection error: {rms:.4f} px")
print(f"focal error: {abs(K_est[0,0] - K_true[0,0]):.3f} px")
print(f"principal-point error: "
      f"{abs(K_est[0,2]-K_true[0,2]):.3f}, {abs(K_est[1,2]-K_true[1,2]):.3f} px")

# ---- Step 5: noisy corners ----
noisy_points = [p + rng.normal(0, 0.3, p.shape).astype(np.float32)
                for p in image_points]
rms_n, K_n, dist_n, _, _ = cv2.calibrateCamera(
    obj_list, noisy_points, (W, H), None, None)
print(f"noisy RMS: {rms_n:.4f} px, noisy focal error: "
      f"{abs(K_n[0,0]-K_true[0,0]):.3f} px")

# ---- Step 6: fresh pose with PnP ----
rvec_gt, tvec_gt = random_pose()
proj_gt, _ = cv2.projectPoints(objp, rvec_gt, tvec_gt, K_est, dist_est)
ok, rvec_pnp, tvec_pnp = cv2.solvePnP(objp, proj_gt, K_est, dist_est)
reproj, _ = cv2.projectPoints(objp, rvec_pnp, tvec_pnp, K_est, dist_est)
err = np.linalg.norm(reproj.reshape(-1, 2) - proj_gt.reshape(-1, 2), axis=1)
print(f"PnP mean reprojection error: {err.mean():.5f} px")

# ---- Step 7: robust PnP with RANSAC ----
corrupt = proj_gt.copy()
corrupt[[3, 17, 40]] += rng.uniform(-50, 50, (3, 1, 2)).astype(np.float64)
ok, rvec_r, tvec_r, inliers = cv2.solvePnPRansac(objp, corrupt, K_est, dist_est)
kept = set(inliers.ravel().tolist())
print(f"RANSAC kept {len(kept)} / {len(objp)} corners")
print(f"corrupted indices in inlier set: {[i for i in (3,17,40) if i in kept]}")

With that workbench closing the loop, the chapter's three-beat arc is complete: Section 12.1 showed how the pinhole model and the intrinsic matrix $K$ project the 3D world into pixels and throw depth away, Sections 12.2 and 12.3 showed how to calibrate that projection for your specific camera, and Section 12.4 showed how to locate the camera every frame with PnP; this section gave the craft that keeps all three honest in production. The single number you have now earned, a calibrated $K$, is what turns each pixel into a metric ray. One ray still cannot tell you depth, but two can: Chapter 13: Two-View Geometry, Stereo & Depth adds a second camera, uses this chapter's intrinsics and distortion to rectify the pair, and finally intersects the rays to win back the dimension that projection destroyed.