Worked Example: A Document Scanner from Scratch

"You photographed an A4 page as a trapezoid at a 40-degree angle, in restaurant lighting, with your thumb in frame. I will fix it. I always fix it. This is my whole personality now."

A Long-Suffering Document Scanner

In about a hundred lines of Python, you are about to rebuild the core of every mobile scanning app on the planet. The textbook taught the theory bottom up: models, coordinates, interpolation, execution, estimation. Real projects run the other way, starting from a goal: the user photographs a paper at an angle; produce a clean, flat, high-contrast scan. This section walks the full distance from goal to working code, including the unglamorous parts (corner ordering, resolution bookkeeping) where real implementations actually break. Figure 5.6.1 shows the route.

1. Stage 1: Find the Page Outline Intermediate

Our scanner's "correspondence problem" is friendlier than Section 5.5's general matching: we know the object of interest is a quadrilateral that contrasts with the background. The classical detection recipe is blur, edge detection, and contour analysis. We downscale first, both for speed and because edge detectors behave more consistently at a standard working resolution; the crucial bookkeeping is to remember the scale factor, because the final warp must run on the full-resolution original. Detection can be lossy; rectification must not be.

# Stage 1 of the scanner: locate the page as the largest convex
# quadrilateral in a downscaled edge map, then scale its corners back to
# full-resolution coordinates so the later warp loses no detail.
import cv2
import numpy as np

PROC_HEIGHT = 600.0          # standard working height for detection

def find_page_quad(image_bgr):
    """Return the page's 4 corners in FULL-RES coords, or None."""
    scale = PROC_HEIGHT / image_bgr.shape[0]
    small = cv2.resize(image_bgr, None, fx=scale, fy=scale,
                       interpolation=cv2.INTER_AREA)

    gray = cv2.cvtColor(small, cv2.COLOR_BGR2GRAY)
    gray = cv2.GaussianBlur(gray, (5, 5), 0)        # tame paper texture
    edges = cv2.Canny(gray, 75, 200)                # edge map
    edges = cv2.dilate(edges, np.ones((3, 3), np.uint8))  # bridge gaps

    contours, _ = cv2.findContours(edges, cv2.RETR_LIST,
                                   cv2.CHAIN_APPROX_SIMPLE)
    contours = sorted(contours, key=cv2.contourArea, reverse=True)[:5]

    for c in contours:                              # biggest first
        peri = cv2.arcLength(c, True)
        approx = cv2.approxPolyDP(c, 0.02 * peri, True)
        if len(approx) == 4 and cv2.isContourConvex(approx):
            if cv2.contourArea(approx) > 0.2 * small.size / 3:
                return approx.reshape(4, 2).astype(np.float64) / scale
    return None

Each ingredient earns its place. The Gaussian blur (from Chapter 3) suppresses paper grain and carpet texture that would otherwise fill the edge map with confetti. Canny, which we use here as a black box and dissect properly in Chapter 9, traces intensity discontinuities; the dilation closes one-pixel gaps in the page border so the contour is a single closed curve. approxPolyDP simplifies each candidate contour with the Douglas-Peucker algorithm at a tolerance of 2 percent of the perimeter: page outlines survive as exactly 4 vertices, while sleeves, mugs, and shadows rarely do. The convexity and minimum-area tests reject the rest.

2. Stage 2: Order the Corners Intermediate

The contour hands us four corners in an arbitrary cyclic order, but getPerspectiveTransform pairs source to destination points by index: if our first destination corner is "top-left", the first source corner had better actually be the page's top-left. Feed the points in a rotated or reflected order and you get a perfectly valid homography to an upside-down or mirror-imaged page. The classic ordering trick uses two scalar functions of each corner $(x, y)$: the sum $x + y$ is smallest at the top-left and largest at the bottom-right; the difference $y - x$ is smallest at the top-right and largest at the bottom-left.

The intuition is that the two diagonals point in perpendicular directions. One scalar (the sum) separates the corners along the main diagonal, the other (the difference) separates them along the anti-diagonal, and between them every corner gets a unique label.

# Stage 2: put the four detected corners into a canonical order so that
# getPerspectiveTransform pairs them with the right destination corners.
# The sum x+y and difference y-x identify each corner's role.
def order_corners(pts):
    """pts: (4, 2) array in any order -> [tl, tr, br, bl]."""
    s = pts.sum(axis=1)            # x + y
    d = np.diff(pts, axis=1)[:, 0] # y - x
    return np.array([pts[np.argmin(s)],    # top-left
                     pts[np.argmin(d)],    # top-right
                     pts[np.argmax(s)],    # bottom-right
                     pts[np.argmax(d)]],   # bottom-left
                    dtype=np.float32)

3. Stage 3: Size the Output and Warp Intermediate

What size should the flattened page be? We measure the quadrilateral's edges in the photo and take the maximum of opposite sides as the output width and height. This preserves as much resolution as the photo captured and gets the aspect ratio approximately right. Only approximately: perspective foreshortening means the photographed side lengths are not the true paper proportions. Recovering the exact aspect ratio of a rectangle from one perspective view is possible, but it requires the camera's focal length, which belongs to the calibration story of Chapter 12. Production apps either do that or simply snap to known paper ratios (A4, Letter); we take the honest approximation.

# Stage 3: size the flat output from the quad's own side lengths, then
# solve the exact 4-point homography and inverse-warp the page to a
# rectangle. Four pairs pin down all eight homography parameters.
def rectify(image_bgr, quad):
    """Warp the quadrilateral region into a flat, axis-aligned scan."""
    tl, tr, br, bl = order_corners(quad)

    W = int(max(np.linalg.norm(br - bl), np.linalg.norm(tr - tl)))
    H = int(max(np.linalg.norm(tr - br), np.linalg.norm(tl - bl)))

    src = np.array([tl, tr, br, bl], dtype=np.float32)
    dst = np.array([[0, 0], [W - 1, 0],
                    [W - 1, H - 1], [0, H - 1]], dtype=np.float32)

    M = cv2.getPerspectiveTransform(src, dst)   # 4 pairs -> 8 DoF, exact
    return cv2.warpPerspective(image_bgr, M, (W, H),
                               flags=cv2.INTER_LINEAR)

Pause on getPerspectiveTransform for a moment, because it closes a loop opened in Section 5.1: a homography has 8 degrees of freedom, each point pair supplies 2 equations, and 4 pairs make the system exactly determined, so the function solves a small linear system and returns the unique homography through our corners. No RANSAC is needed here, unlike Section 5.5, because we have exactly four correspondences and trust all of them; the robustness lives upstream in the contour tests. The warp call then runs the inverse-mapping gather of Section 5.4 with the bilinear kernel of Section 5.3.

4. Stage 4: Binarize Like a Scanner Beginner

A geometric rectangle of a photo still looks like a photo: gray paper, uneven lighting, a shadow from your hand. The "scanned document" look is a thresholding problem, and the right tool is the adaptive thresholding of Chapter 2, which computes a local threshold per neighborhood and therefore shrugs off illumination gradients that destroy any single global threshold:

# Stage 4 plus the main program: adaptive thresholding gives the crisp
# black-on-white scanner look despite uneven lighting, and the bottom
# block chains detection, rectification, and binarization end to end.
def to_scan(image_bgr):
    gray = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2GRAY)
    return cv2.adaptiveThreshold(gray, 255,
                                 cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
                                 cv2.THRESH_BINARY,
                                 blockSize=21, C=10)

# The complete scanner, end to end:
image = cv2.imread("receipt_photo.jpg")
quad = find_page_quad(image)
if quad is None:
    raise SystemExit("No document found: check contrast with background")
flat = rectify(image, quad)
scan = to_scan(flat)
cv2.imwrite("scan.png", scan)
print(f"saved {scan.shape[1]}x{scan.shape[0]} scan")

saved 1187x1684 scan

And that is the entire scanner: roughly one hundred lines including comments, no machine learning, latency dominated by the single full-resolution warp. The binarized output often shows speckle noise from paper texture and dust; cleaning that up with a morphological opening is literally the first worked example of the next chapter, which picks up this exact image.

# imutils folds corner ordering, output sizing, and the perspective warp
# (our order_corners plus rectify) into a single tested call; detection
# and binarization stay in your hands as the per-application stages.
from imutils.perspective import four_point_transform
flat = four_point_transform(image, quad.reshape(4, 2))

5. Failure Modes and Hardening Advanced

Turning this demo into a product is mostly about the inputs that break it. Four failure classes account for nearly everything, and each maps to a specific upgrade path:

• Low edge contrast. White paper on a white desk gives Canny nothing. Mitigations: try multiple threshold pairs, run detection per color channel and on a saturation channel, or fall back to asking the user to tap the corners. The durable fix is replacing stage 1 with a segmentation model, the approach of Chapter 24, which is exactly what modern phone scanners do.
• Distractor quadrilaterals. Laptops, monitors, and floor tiles are large convex quads. Mitigations: prefer the quad containing the image center, score candidates by text-like high-frequency content inside, or track stability across video frames.
• Non-planar pages. Books near the spine, curled receipts. The homography is structurally wrong; see the practical example and research frontier.
• Extreme angles. Beyond roughly 60 degrees of tilt, the far edge's effective resolution collapses; the warp magnifies a few hundred captured pixels into a thousand output pixels of mush. Detect by comparing opposite side lengths and prompt for a re-shoot; no interpolation from Section 5.3 can manufacture detail the sensor never sampled.

6. What This Project Taught Beginner

Walk back through the hundred lines and notice how the chapter's sections each carried a stage: the hierarchy (5.1) told us a photographed plane needs exactly a homography, no more, no less; homogeneous coordinates (5.2) are why getPerspectiveTransform returns a 3×3 matrix and why the warp divides by $W$; interpolation (5.3) fills every output pixel from fractional source positions; inverse mapping (5.4) is the reason the output has no holes; and the four corners are a tiny, trusted correspondence set, the same currency 5.5 earned with feature matching and RANSAC. One pipeline, five ideas, each load-bearing.

The scanner also hands the book its next problem. Its output is a binary image, and binary images have their own algebra: erosion to strip speckle, dilation to heal broken strokes, connected components to find characters, shape descriptors to classify them. That algebra is Chapter 6: Morphology, Binary Images & Shape, and it begins exactly where scan.png ends.

The scanner exercised the chapter on a single image with four trusted corners. The chapter's other headline application, stitching, needs the full registration machinery of Section 5.5 running on two images at once. Build it yourself in the Hands-On Lab below: a two-photo panorama stitcher that detects and matches features, fits a homography with RANSAC, and warps one frame onto the other's canvas. It is the natural companion to the scanner and the capstone for everything in Chapter 5.

import cv2
import numpy as np

left  = cv2.imread("left.jpg")    # will be warped onto the right
right = cv2.imread("right.jpg")   # the fixed reference frame

gray_l = cv2.cvtColor(left,  cv2.COLOR_BGR2GRAY)
gray_r = cv2.cvtColor(right, cv2.COLOR_BGR2GRAY)

# TODO: create an ORB detector aiming for a few thousand features
# Hint: cv2.ORB_create(nfeatures=...)
orb = ...

# TODO: detect keypoints and compute descriptors for BOTH gray images
# Hint: orb.detectAndCompute(image, None) returns (keypoints, descriptors)
k_l, d_l = ...
k_r, d_r = ...
print(f"left: {len(k_l)} kp, right: {len(k_r)} kp")

# TODO: build a brute-force matcher with Hamming distance and crossCheck=True
# Hint: cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
bf = ...

# TODO: match d_l against d_r, then sort matches by ascending distance
# and keep the best few hundred
matches = ...
matches = sorted(matches, key=lambda m: m.distance)[:600]
print(f"{len(matches)} candidate matches")

src = np.float32([k_l[m.queryIdx].pt for m in matches]).reshape(-1, 1, 2)
dst = np.float32([k_r[m.trainIdx].pt for m in matches]).reshape(-1, 1, 2)

# TODO: estimate the homography mapping left points to right points,
# using RANSAC with a reprojection threshold of about 4 pixels
# Hint: cv2.findHomography(src, dst, cv2.RANSAC, ransacReprojThreshold=...)
H, mask = ...

n_in = int(mask.sum())
print(f"RANSAC kept {n_in}/{len(matches)} matches as inliers")
assert n_in >= 30, "too few inliers: re-shoot with more overlap"

h_l, w_l = left.shape[:2]
h_r, w_r = right.shape[:2]

corners_l = np.float32([[0, 0], [0, h_l], [w_l, h_l], [w_l, 0]]).reshape(-1, 1, 2)
warped_corners = cv2.perspectiveTransform(corners_l, H)
corners_r = np.float32([[0, 0], [0, h_r], [w_r, h_r], [w_r, 0]]).reshape(-1, 1, 2)
all_corners = np.concatenate([warped_corners, corners_r], axis=0)

x_min, y_min = np.int32(all_corners.min(axis=0).ravel() - 0.5)
x_max, y_max = np.int32(all_corners.max(axis=0).ravel() + 0.5)

# TODO: build the 3x3 translation matrix T that shifts by (-x_min, -y_min)
# Hint: a homogeneous translation is np.array([[1,0,tx],[0,1,ty],[0,0,1]])
T = ...
canvas_size = (x_max - x_min, y_max - y_min)

# TODO: warp the left image onto the canvas using T @ H as the transform
# Hint: cv2.warpPerspective(left, T @ H, canvas_size)
pano = ...

# Paste the right image at its translated location (overwrite, for now)
tx, ty = int(T[0, 2]), int(T[1, 2])
pano[ty:ty + h_r, tx:tx + w_r] = right

cv2.imwrite("panorama_hard_seam.jpg", pano)
print(f"panorama size: {pano.shape[1]}x{pano.shape[0]}")

warped_left = cv2.warpPerspective(left, T @ H, canvas_size)
right_layer = np.zeros_like(warped_left)
right_layer[ty:ty + h_r, tx:tx + w_r] = right

# Coverage masks (single channel) for each layer
mask_l = (warped_left.sum(axis=2) > 0).astype(np.float32)
mask_r = (right_layer.sum(axis=2) > 0).astype(np.float32)

# TODO: turn the two binary masks into normalized blend weights so that
# wl + wr = 1 wherever either image has content (avoid divide-by-zero)
# Hint: total = mask_l + mask_r; weight = mask / np.maximum(total, 1e-6)
wl = ...
wr = ...

blended = (warped_left * wl[..., None] + right_layer * wr[..., None])
cv2.imwrite("panorama.jpg", blended.astype(np.uint8))

# Two-photo panorama stitcher: detect, match, RANSAC-fit, warp, blend.
import cv2
import numpy as np

def stitch(left, right):
    gray_l = cv2.cvtColor(left,  cv2.COLOR_BGR2GRAY)
    gray_r = cv2.cvtColor(right, cv2.COLOR_BGR2GRAY)

    # Step 1: detect ORB features in both images
    orb = cv2.ORB_create(nfeatures=4000)
    k_l, d_l = orb.detectAndCompute(gray_l, None)
    k_r, d_r = orb.detectAndCompute(gray_r, None)

    # Step 2: match with Hamming distance + cross-check, keep best 600
    bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
    matches = sorted(bf.match(d_l, d_r), key=lambda m: m.distance)[:600]

    # Step 3: homography from left to right, RANSAC rejecting outliers
    src = np.float32([k_l[m.queryIdx].pt for m in matches]).reshape(-1, 1, 2)
    dst = np.float32([k_r[m.trainIdx].pt for m in matches]).reshape(-1, 1, 2)
    H, mask = cv2.findHomography(src, dst, cv2.RANSAC, ransacReprojThreshold=4.0)
    n_in = int(mask.sum())
    print(f"RANSAC kept {n_in}/{len(matches)} matches as inliers")
    assert n_in >= 30, "too few inliers: re-shoot with more overlap"

    # Step 4: size the canvas and build a translation offset
    h_l, w_l = left.shape[:2]
    h_r, w_r = right.shape[:2]
    corners_l = np.float32([[0, 0], [0, h_l], [w_l, h_l], [w_l, 0]]).reshape(-1, 1, 2)
    warped_corners = cv2.perspectiveTransform(corners_l, H)
    corners_r = np.float32([[0, 0], [0, h_r], [w_r, h_r], [w_r, 0]]).reshape(-1, 1, 2)
    all_corners = np.concatenate([warped_corners, corners_r], axis=0)
    x_min, y_min = np.int32(all_corners.min(axis=0).ravel() - 0.5)
    x_max, y_max = np.int32(all_corners.max(axis=0).ravel() + 0.5)
    T = np.array([[1, 0, -x_min],
                  [0, 1, -y_min],
                  [0, 0, 1]], dtype=np.float64)
    canvas_size = (x_max - x_min, y_max - y_min)

    # Step 5: warp left onto the canvas; stage right on its own layer
    warped_left = cv2.warpPerspective(left, T @ H, canvas_size)
    tx, ty = int(T[0, 2]), int(T[1, 2])
    right_layer = np.zeros_like(warped_left)
    right_layer[ty:ty + h_r, tx:tx + w_r] = right

    # Step 6: feathered linear blend across the overlap
    mask_l = (warped_left.sum(axis=2) > 0).astype(np.float32)
    mask_r = (right_layer.sum(axis=2) > 0).astype(np.float32)
    mask_l = cv2.GaussianBlur(mask_l, (0, 0), sigmaX=25)
    mask_r = cv2.GaussianBlur(mask_r, (0, 0), sigmaX=25)
    total = mask_l + mask_r
    wl = mask_l / np.maximum(total, 1e-6)
    wr = mask_r / np.maximum(total, 1e-6)
    blended = warped_left * wl[..., None] + right_layer * wr[..., None]
    return blended.astype(np.uint8)

if __name__ == "__main__":
    left  = cv2.imread("left.jpg")
    right = cv2.imread("right.jpg")
    pano = stitch(left, right)
    cv2.imwrite("panorama.jpg", pano)
    print(f"panorama size: {pano.shape[1]}x{pano.shape[0]}")