Datasets & Benchmarks for Geometry, Flow & Tracking

"They gave me ground truth from a laser scanner and a withheld test server. I cannot overfit to a number I am not allowed to see. It is the kindest thing a benchmark ever did for me."

An Algorithm With Nowhere to Cheat

The pipelines of Section 17.2 hand back numbers, a registered-image count, a reprojection error, a recovered trajectory, and a number means nothing until you can say what it would score against an agreed reference. That is the third rung of the chapter's verbs-pipelines-scoreboards ladder, and it is where this section lives. Every chapter in Part II quietly relied on a benchmark to justify its claims: the matchers of Chapter 10, the stereo of Chapter 13, the SLAM of Chapter 14, the flow and tracking of Chapter 15. This section gathers those proving grounds into one reference, organized by task, so that before you trust any result, yours or a paper's, you know which dataset it was measured on, what metric was reported, and what that metric actually rewards. We move task by task, from features to tracking.

1. Calibration Targets & Feature Matching Beginner

The simplest "dataset" in Part II is a printed pattern. Camera calibration (Chapter 12) needs a target of known geometry, and the choice of target affects accuracy. The classic checkerboard is detected by cv2.findChessboardCorners; ChArUco boards (a checkerboard fused with ArUco markers) tolerate occlusion and partial views, which is why they have become the modern default for serious calibration. For feature matching, the standard benchmark is HPatches (Balntas et al., CVPR 2017): 116 scenes, each a reference image plus five images related by a known homography, split into illumination and viewpoint changes, with descriptor matching scored by mean average precision (mAP), a single number summarizing how well the correct matches are ranked above the wrong ones across all the test pairs. It is the dataset that lets you say "descriptor A beats descriptor B" with a number rather than an opinion.

2. Stereo & Multi-View Depth Intermediate

Dense depth (Chapter 13) has three canonical benchmarks, each contributing a different kind of difficulty. Middlebury Stereo is the historical reference: controlled indoor scenes, structured-light ground truth, and sub-pixel disparity evaluation that exposes fine errors. KITTI Stereo (Geiger et al., CVPR 2012) adds outdoor driving difficulty, real sensor noise, and LiDAR-derived sparse ground truth, with a withheld online test server. ETH3D (Schöps et al., CVPR 2017) supplies high-resolution multi-view stereo with laser-scan ground truth, the modern standard for reconstruction-quality evaluation, and the benchmark the COLMAP-class tools of Section 17.2 are judged on.

The standard stereo metric on KITTI is D1: the percentage of pixels whose disparity error exceeds both 3 pixels and 5 percent of the true disparity. Code 17.3.1 implements it directly so the definition is unambiguous, masking the invalid pixels where sparse LiDAR ground truth is missing.

# KITTI D1 stereo metric: count the fraction of valid pixels whose
# disparity error exceeds both an absolute (3 px) and a relative (5%)
# threshold, skipping pixels with no LiDAR ground truth.
import numpy as np

def d1_error(pred_disp, gt_disp, tau_px=3.0, tau_rel=0.05):
    """KITTI D1: fraction of valid pixels with disparity error too large.
    A pixel is an outlier if abs error > 3 px AND > 5% of true disparity."""
    valid = gt_disp > 0                       # LiDAR ground truth is sparse
    err = np.abs(pred_disp - gt_disp)
    outlier = (err > tau_px) & (err > tau_rel * gt_disp)
    return 100.0 * np.count_nonzero(outlier & valid) / np.count_nonzero(valid)

# Lower is better; strong stereo methods report D1 around 1-2% on KITTI.
print(f"D1 = {d1_error(pred, gt):.2f}%")

D1 = 1.83%

3. Optical Flow Intermediate

Optical flow (Chapter 15) is graded by end-point error (EPE): the average Euclidean distance between the predicted flow vector and the ground-truth flow vector, per pixel. For a predicted flow $(u, v)$ and ground truth $(u^\*, v^\*)$,

$$ \text{EPE} = \frac{1}{N} \sum_{\text{pixels}} \sqrt{(u - u^\*)^2 + (v - v^\*)^2}. $$

The two anchor datasets are complementary, and a serious flow paper reports both. MPI Sintel (Butler et al., ECCV 2012) is rendered from an open-source animated film: it provides dense, pixel-perfect ground truth and deliberately hard conditions, large motions, motion blur, atmospheric haze, and a "final" pass with realistic rendering artifacts. KITTI Flow supplies real driving scenes with sparse LiDAR-derived ground truth, the reality check that synthetic Sintel cannot give. The 2023 Spring benchmark (Mehl et al., CVPR 2023) extends the synthetic line to high resolution with extremely dense ground truth, addressing the saturation of Sintel by the best learned methods. Table 17.3.1 places the flow and depth datasets side by side.

4. SLAM, Odometry & Tracking Advanced

Trajectory estimation (Chapter 14) is evaluated against a recorded ground-truth path. The two standard metrics are Absolute Trajectory Error (ATE), which measures global consistency after rigidly aligning the estimated and true trajectories, and Relative Pose Error (RPE), which measures local drift over fixed time intervals. The canonical datasets are TUM RGB-D (Sturm et al., IROS 2012), which standardized the ATE/RPE file formats, the older KITTI Odometry split for driving, and the large synthetic TartanAir (Wang et al., IROS 2020) for hard photometric and motion conditions. The alignment step is the subtle part, and getting it wrong is the most common evaluation error in SLAM. The illustration below dramatizes the worst version of that error, the scale-correction trap detailed in this section's Practical Example.

Multi-object tracking (Chapter 15) has its own benchmark and a metric debate worth knowing. The MOTChallenge suite (Dendorfer et al., IJCV 2021) is the standard for pedestrian tracking. Its historical headline metric, Multiple Object Tracking Accuracy (MOTA), combines false positives, missed detections, and identity switches, but it is dominated by detection quality and barely penalizes identity errors. The newer Higher Order Tracking Accuracy (HOTA) metric (Luiten et al., IJCV 2021) explicitly balances detection accuracy against association accuracy by combining them as a geometric mean, and has become the preferred number precisely because it rewards the thing tracking is actually about: keeping the right identity on the right object over time. The geometric mean is the active ingredient: because $\sqrt{a \cdot b}$ collapses toward zero whenever either factor is small, a tracker cannot buy a high HOTA by acing detection while botching association (or the reverse), the way an arithmetic mean would let it average a strong half against a weak one. Only a method strong on both terms scores well, which is exactly the balance MOTA's additive bookkeeping fails to enforce. Figure 17.3.1 shows why the two metrics can rank the same trackers differently.

The alignment-and-scale subtlety above is not a detail to read past; it is the single most common way an honest-looking trajectory number lies. The best way to internalize it is to build the scoreboard yourself once, then watch a tool reproduce it. Put the section's evaluation lessons into practice in the Hands-On Lab at the end of this section, which has you implement Absolute Trajectory Error from scratch, expose the scale-correction trap on a controlled example, and finish by reproducing your own numbers with evo.

pip install numpy evo

import numpy as np

rng = np.random.default_rng(0)
t = np.linspace(0, 4 * np.pi, 200)
# Ground-truth path: a rising spiral, positions only (N x 3), in meters.
gt = np.stack([np.cos(t), np.sin(t), 0.15 * t], axis=1)

# TODO: build `est` from `gt` as a monocular estimate would relate to truth:
#   1. shrink by a true scale factor s_true = 0.5 (monocular has no metric scale),
#   2. rotate by a fixed rotation R_true about the z axis (e.g. 30 degrees),
#   3. translate by a fixed offset t_true (e.g. [2, -1, 0.5]),
#   4. add small Gaussian noise (std 0.01) so the fit is not exact.
# Hint: est_i = s_true * (R_true @ gt_i) + t_true + noise_i
s_true = 0.5
# ... define R_true, t_true, then est = ...
print("gt shape", gt.shape, "est shape", est.shape)

def umeyama(src, dst, with_scale):
    """Least-squares similarity (s, R, t) mapping src onto dst.
    src, dst: (N, 3) arrays. Returns s (float), R (3x3), t (3,)."""
    mu_src, mu_dst = src.mean(0), dst.mean(0)
    sc, dc = src - mu_src, dst - mu_dst
    cov = (dc.T @ sc) / len(src)
    U, D, Vt = np.linalg.svd(cov)
    S = np.eye(3)
    if np.linalg.det(U) * np.linalg.det(Vt) < 0:   # reflection guard
        S[2, 2] = -1
    R = U @ S @ Vt
    # TODO: compute the scale `s`.
    #   with_scale=True  -> s = trace(diag(D) @ S) / variance of src
    #   with_scale=False -> s = 1.0 (rigid alignment, no rescaling)
    # Then compute t = mu_dst - s * R @ mu_src and return s, R, t.
    ...
    return s, R, t

def ate_rmse(gt, est, with_scale):
    # TODO: align est onto gt with umeyama(est, gt, with_scale),
    #   map every est point: est_aligned = s * (R @ est_i) + t,
    #   then return sqrt(mean(||gt_i - est_aligned_i||^2)) and the scale s.
    ...
    return rmse, s

rmse_rigid, s_rigid = ate_rmse(gt, est, with_scale=False)
rmse_sim,   s_sim   = ate_rmse(gt, est, with_scale=True)
print(f"ATE rigid  (no scale): {rmse_rigid:.4f} m, recovered scale {s_rigid:.3f}")
print(f"ATE similarity (-s):   {rmse_sim:.4f} m, recovered scale {s_sim:.3f}")

# TODO: print a one-line verdict. The similarity ATE looks excellent only
#   because it silently rescaled the estimate by ~2x to match metric truth.
#   The rigid ATE is the number a robot WITHOUT a scale oracle actually lives with.
print(f"similarity hides a {1.0 / s_sim:.2f}x scale error that rigid alignment exposes")
# Expected: rigid ATE >> similarity ATE; recovered scale ~ 0.5.

# TUM format per line: timestamp tx ty tz qx qy qz qw  (identity quaternion here).
def write_tum(path, xyz):
    with open(path, "w") as f:
        for i, p in enumerate(xyz):
            f.write(f"{i:.6f} {p[0]} {p[1]} {p[2]} 0 0 0 1\n")

write_tum("gt.txt", gt)
write_tum("est.txt", est)
# TODO: from the shell, run both and compare to your Step 3 numbers:
#   evo_ape tum gt.txt est.txt -a        # rigid: should match rmse_rigid
#   evo_ape tum gt.txt est.txt -a -s     # similarity: should match rmse_sim
print("wrote gt.txt and est.txt; run evo_ape to confirm")

import numpy as np

# ---- Step 1: ground truth and a flawed monocular estimate ----
rng = np.random.default_rng(0)
t = np.linspace(0, 4 * np.pi, 200)
gt = np.stack([np.cos(t), np.sin(t), 0.15 * t], axis=1)

s_true = 0.5
th = np.deg2rad(30)
R_true = np.array([[np.cos(th), -np.sin(th), 0],
                   [np.sin(th),  np.cos(th), 0],
                   [0, 0, 1]])
t_true = np.array([2.0, -1.0, 0.5])
est = s_true * (R_true @ gt.T).T + t_true + rng.normal(0, 0.01, gt.shape)

# ---- Step 2: Umeyama similarity alignment ----
def umeyama(src, dst, with_scale):
    mu_src, mu_dst = src.mean(0), dst.mean(0)
    sc, dc = src - mu_src, dst - mu_dst
    cov = (dc.T @ sc) / len(src)
    U, D, Vt = np.linalg.svd(cov)
    S = np.eye(3)
    if np.linalg.det(U) * np.linalg.det(Vt) < 0:
        S[2, 2] = -1
    R = U @ S @ Vt
    if with_scale:
        var = (sc ** 2).sum() / len(src)
        s = np.trace(np.diag(D) @ S) / var
    else:
        s = 1.0
    t = mu_dst - s * R @ mu_src
    return s, R, t

# ---- Step 3: ATE after alignment ----
def ate_rmse(gt, est, with_scale):
    s, R, t = umeyama(est, gt, with_scale)
    est_aligned = (s * (R @ est.T)).T + t
    rmse = np.sqrt(((gt - est_aligned) ** 2).sum(1).mean())
    return rmse, s

rmse_rigid, s_rigid = ate_rmse(gt, est, with_scale=False)
rmse_sim,   s_sim   = ate_rmse(gt, est, with_scale=True)
print(f"ATE rigid  (no scale): {rmse_rigid:.4f} m, recovered scale {s_rigid:.3f}")
print(f"ATE similarity (-s):   {rmse_sim:.4f} m, recovered scale {s_sim:.3f}")

# ---- Step 4: read the trap ----
print(f"similarity hides a {1.0 / s_sim:.2f}x scale error that rigid alignment exposes")

# ---- Step 5: cross-check against evo ----
def write_tum(path, xyz):
    with open(path, "w") as f:
        for i, p in enumerate(xyz):
            f.write(f"{i:.6f} {p[0]} {p[1]} {p[2]} 0 0 0 1\n")

write_tum("gt.txt", gt)
write_tum("est.txt", est)
print("wrote gt.txt and est.txt")
# Shell cross-check:
#   evo_ape tum gt.txt est.txt -a       -> rmse matches rmse_rigid
#   evo_ape tum gt.txt est.txt -a -s    -> rmse matches rmse_sim, scale ~ 0.5