From Disparity to Depth Maps

"At one meter I am a precision instrument. At ten meters I am an educated guess. At fifty meters I am astrology with a lens cap. The formula is right there; nobody reads the formula."

A Stereo Rig Spread Slightly Too Thin

Everything in this chapter so far produced relationships: constraints, matrices, pixel shifts. This is the section where numbers with units finally appear, the meters that a robot can brake on, and where the innocuous-looking baseline $B$, measured with a ruler during the calibration of Chapter 12, resolves the scale ambiguity that haunted Section 13.2's essential matrix.

1. Similar Triangles, One Formula Beginner

Take the rectified geometry of Section 13.4: two identical cameras with focal length $f$ (in pixels), optical centers separated by baseline $B$ (in meters), axes parallel. A point at depth $Z$ projects at $x_L = f\,X/Z$ in the left image and $x_R = f\,(X - B)/Z$ in the right. Subtract:

$$ d \;=\; x_L - x_R \;=\; \frac{fB}{Z} \qquad\Longleftrightarrow\qquad Z \;=\; \frac{fB}{d}. $$

Depth is inversely proportional to disparity, with the proportionality constant $fB$ set entirely by the rig. The geometry behind the algebra is two similar triangles, drawn in Figure 13.5.1 alongside the formula's most important consequence. Sanity-check the numbers on a typical robotics rig ($f = 700$ px, $B = 12$ cm): a pixel at $d = 84$ sits at $Z = 0.12 \cdot 700 / 84 = 1.0$ m; at $d = 8.4$, ten meters; at $d = 0$, infinity. The whole range from ten meters to the horizon is compressed into the last eight disparity levels, while the first meter enjoys dozens. Disparity is a depth scale that lavishes resolution on the nearby.

2. The Quadratic Error Law Advanced

The matcher of Section 13.4 delivers disparity with some uncertainty $\Delta d$, typically a quarter pixel or so after subpixel refinement. Differentiating $Z = fB/d$ with respect to $d$ and substituting $d = fB/Z$ gives the error propagation:

$$ \Delta Z \;=\; \frac{fB}{d^2}\,\Delta d \;=\; \frac{Z^2}{fB}\,\Delta d. $$

Read it twice; it is the most operationally important equation in the chapter. Depth error grows with the square of depth, shrinks linearly with baseline and focal length, and scales directly with matcher noise. For the rig above ($fB = 84$ m-px) at $\Delta d = 0.25$ px: $\pm 3$ mm at 1 m, $\pm 7.4$ cm at 5 m, $\pm 30$ cm at 10 m, $\pm 1.9$ m at 25 m. The same quarter-pixel of matcher noise spans two orders of magnitude in metric damage, which is why "what is your stereo's accuracy?" has no answer without "at what range?".

The formula also writes the design manual. Want accuracy at range? Increase $fB$: a wider baseline or a longer lens. Both purchases bring costs the formula does not show. A wider baseline reduces the overlap between views, increases occlusion (more pixels seen by only one camera, recall Section 13.4's invalid halos), makes matching harder because perspectives differ more, and raises the minimum measurable depth, since nearby points exceed the matcher's disparity range $d_{\max}$: $Z_{\min} = fB/d_{\max}$. A longer focal length trades field of view away. Every stereo product is a frozen argument among these pressures: a 6 cm phone-camera pair optimizes portrait distance, a 12 cm robot rig the 0.5 to 8 m workspace, a 1 m automotive rig the highway.

3. From Disparity Map to Metric Point Cloud Intermediate

The per-pixel conversion could be hand-rolled from $Z = fB/d$ plus the pinhole back-projection of Chapter 12 ($X = (x - c_x)Z/f$, $Y = (y - c_y)Z/f$), but rectification already packaged the whole affair into the $4 \times 4$ reprojection matrix $Q$ from cv2.stereoRectify: in homogeneous coordinates, $(x, y, d, 1)^\top$ maps through $Q$ to the 3D point. cv2.reprojectImageTo3D applies it to the entire map at once. The code below converts the SGBM output of Section 13.4 into a depth map and a colored PLY point cloud you can open in MeshLab or CloudCompare:

import cv2
import numpy as np

# disp (float32 pixels) and Q from Sections 13.4 / 13.1's rig setup
points = cv2.reprojectImageTo3D(disp, Q)        # HxWx3, in calibration units
Z = points[:, :, 2]

valid = (disp > disp.min()) & np.isfinite(Z) & (Z > 0) & (Z < 20.0)
print(f"median depth: {np.median(Z[valid]):.2f} m, "
      f"valid pixels: {100 * valid.mean():.1f}%")
# median depth: 3.41 m, valid pixels: 88.6%

# per-pixel uncertainty from the quadratic law (delta_d = 0.25 px)
fB = Q[2, 3] / Q[3, 2] if Q[3, 2] != 0 else None   # or f * B directly
sigma_Z = (Z ** 2) * 0.25 / abs(fB)
print(f"sigma at median depth: {np.median(sigma_Z[valid]) * 100:.1f} cm")

# export a colored point cloud for MeshLab / CloudCompare
colors = cv2.cvtColor(rectL, cv2.COLOR_BGR2RGB)
xyz, rgb = points[valid], colors[valid]
with open("cloud.ply", "w") as fh:
    fh.write("ply\nformat ascii 1.0\n"
             f"element vertex {len(xyz)}\n"
             "property float x\nproperty float y\nproperty float z\n"
             "property uchar red\nproperty uchar green\nproperty uchar blue\n"
             "end_header\n")
    for (X, Y, Zp), (r, g, b) in zip(xyz, rgb):
        fh.write(f"{X:.4f} {Y:.4f} {Zp:.4f} {r} {g} {b}\n")

Three hygiene rules keep the output trustworthy. Mask aggressively: invalid disparities reproject to infinities and the matcher's mistakes become dramatic "flying pixels" stretched along rays; the depth gate in the mask (here 20 m) should encode where the quadratic law says your rig stops being credible, not where measurements stop existing. Mind the units: $Q$ is in whatever units the calibration's $T$ used; calibrate in meters or convert deliberately. Carry uncertainty: the $\sigma_Z$ computed above costs two lines and turns a pretty cloud into a measurement, letting downstream consumers (grasp planners, mapping, the fusion filters of Chapter 15) weight near points more than far ones, as they should.

4. Stereo Among Its Rivals Intermediate

Passive stereo is one of several ways to buy a depth map, and honest engineering compares them. Active stereo (Intel RealSense D4xx and kin) adds an infrared dot projector to a stereo pair: the matcher is this chapter's, but the texture problem of Section 13.4 is solved by carrying texture along, at the cost of range (the pattern washes out in sunlight and beyond a few meters). Structured light (the original Kinect, industrial scanners) replaces the second camera with the projector and triangulates against the known pattern: superb close-range precision, same sunlight allergy. Time-of-flight cameras and LiDAR measure photon round-trip time directly: error grows roughly linearly with range rather than quadratically, which is why long-range autonomy leans on them, at the price of cost, resolution, and interference. And learned monocular depth needs only one camera and a network, predicting plausible depth from pictorial cues alone; its catch is the affine ambiguity (relative depth up to unknown scale and shift) and a confident wrongness on scenes unlike its training data. Stereo's standing offer: metric, dense, passive, cheap, with a precision profile you can compute from $f$, $B$, and $\Delta d$ before buying a single part.