Image Formats & Compression: PNG, JPEG & WebP

"I divided the image into 8 by 8 blocks and threw away what nobody would miss. Apparently somebody missed it."

A Remorseful JPEG Quantizer

This section is where the whole chapter cashes in. Section 1.4 gave us YCbCr and chroma subsampling; Section 1.2 gave us quantization; Section 1.3 gave us bit depth; and the frequency-domain view that completes the picture is coming in Chapter 4. JPEG is precisely these ideas composed into one pipeline, which is why we can now read it end to end. But first, the cleaner half of the world: formats that throw nothing away.

1. Lossless Storage: PNG Beginner

A lossless format guarantees that decode(encode(image)) equals the original array exactly. PNG, the workhorse, achieves compression in two stages: first a per-row prediction filter (each scanline is rewritten as differences from already-decoded neighbors, using one of five predictors chosen per row), then the generic DEFLATE compressor over the residuals. Prediction is what makes it work on images: natural rows resemble their neighbors, so residuals are small and highly compressible. PNG supports 8-bit and 16-bit channels, grayscale, RGB, indexed palettes, and an alpha channel, which makes it the default answer for anything synthetic or anything that is secretly data: charts, UI screenshots, depth maps, and above all annotation masks.

Code 1.5.1 demonstrates the single most important format rule in this book with a 12-line experiment: round-trip a segmentation label mask through JPEG and through PNG, and count how many labels change.

import cv2
import numpy as np

# A binary segmentation label: class 1 inside a square, class 0 outside.
mask = np.zeros((128, 128), np.uint8)
mask[32:96, 32:96] = 1

# Round trip through JPEG (stored scaled to 0/255, as masks often are).
ok, buf = cv2.imencode(".jpg", mask * 255, [cv2.IMWRITE_JPEG_QUALITY, 90])
back = cv2.imdecode(buf, cv2.IMREAD_GRAYSCALE)
relabeled = (back > 127).astype(np.uint8)
print("JPEG round trip: labels changed at",
      int((relabeled != mask).sum()), "pixels")

# Round trip through PNG.
ok, buf = cv2.imencode(".png", mask)
back_png = cv2.imdecode(buf, cv2.IMREAD_GRAYSCALE)
print("PNG  round trip: labels changed at",
      int((back_png != mask).sum()), "pixels")

JPEG round trip: labels changed at 96 pixels
PNG  round trip: labels changed at 0 pixels

2. JPEG: This Chapter, Composed Into a Codec Intermediate

JPEG earns its 10:1 to 20:1 compression by spending the chapter's entire toolkit against human perception. Figure 1.5.1 traces the encoder; every stage should now look familiar.

Reading Figure 1.5.1 stage by stage: the encoder first converts to YCbCr and discards three quarters of the chroma samples (Code 1.4.4 showed how little that costs). Each channel is then tiled into 8×8 blocks, and each block passes through the discrete cosine transform, rewriting 64 pixel values as 64 frequency coefficients: one average (DC) plus progressively finer patterns. Perceptual knowledge enters at the quantization step: each coefficient is divided by an entry from a quantization table and rounded to an integer. The table's entries are large for high frequencies (which the eye barely sees, so they round to zero in droves) and small for low frequencies. The quality parameter simply scales this table. Everything after rounding (zigzag ordering, run-length and Huffman coding) is lossless bookkeeping that exploits all those zeros. The artifacts follow directly: blocking, because each 8×8 tile is quantized independently and neighbors disagree at the seam; ringing near sharp edges, because an edge needs high frequencies that got rounded away; and color bleeding, from the subsampled chroma.

3. Measuring the Damage: PSNR and SSIM Intermediate

Format debates end when you measure. The bluntest instrument is mean squared error dressed in decibels, the peak signal-to-noise ratio:

with $b = 8$ for ordinary images. PSNR is three lines of NumPy, monotone in MSE, and correlates only loosely with what humans see: it penalizes a brightness shift heavily while shrugging at smeared texture. The structural similarity index (SSIM) was designed to do better by comparing local luminance, contrast, and structure between windows of the two images:

where $\mu$, $\sigma^2$, and $\sigma_{xy}$ are windowed means, variances, and covariance, and $C_1, C_2$ stabilize the ratios. SSIM lives in $[-1, 1]$ with 1 meaning identical. These two metrics open an arc that runs the length of this book: from judging pixels here, to judging detections with IoU and mAP in Chapter 23, to judging entire generated distributions with FID in Chapter 37. Code 1.5.2 sweeps JPEG quality and measures both.

import cv2
import numpy as np
from skimage.metrics import peak_signal_noise_ratio, structural_similarity

# Self-contained test image: smooth waves + hard edges + fine texture,
# the three content types compression treats most differently.
rng = np.random.default_rng(11)
h, w = 384, 512
yy, xx = np.mgrid[0:h, 0:w].astype(np.float32)
img = np.dstack([
    120 + 90 * np.sin(xx / 40) * np.cos(yy / 60),        # smooth waves
    np.where((xx // 64 + yy // 64) % 2 == 0, 200, 60),   # checker edges
    rng.normal(128, 30, (h, w)),                         # noise texture
]).clip(0, 255).astype(np.uint8)

print(" qual    bytes  PSNR(dB)   SSIM")
for q in [95, 75, 50, 25, 10]:
    ok, buf = cv2.imencode(".jpg", img, [cv2.IMWRITE_JPEG_QUALITY, q])
    dec = cv2.imdecode(buf, cv2.IMREAD_COLOR)
    p = peak_signal_noise_ratio(img, dec)
    s = structural_similarity(img, dec, channel_axis=2)
    print(f"  q{q:<3} {len(buf):>8}    {p:5.2f}  {s:.4f}")

 qual    bytes  PSNR(dB)   SSIM
  q95   227004    38.74  0.9821
  q75   105177    33.62  0.9384
  q50    72436    31.49  0.9046
  q25    48327    29.32  0.8531
  q10    26358    26.05  0.7402

4. WebP and the Modern Format Landscape Intermediate

WebP, derived from the VP8 video codec's intra-frame coder, improves on JPEG by predicting each block from its decoded neighbors before transforming the residual, typically saving 25 to 35% of bytes at matched quality; it also offers a genuinely lossless mode (a different algorithm entirely) and alpha support in both modes. Its successors push further: AVIF applies the AV1 codec's intra tools and shines at low bitrates; JPEG XL targets both worlds with high-quality lossy compression, fast lossless mode, and the party trick of losslessly re-encoding legacy JPEGs about 20% smaller. Code 1.5.3 compares the three formats OpenCV writes natively.

for ext, params in [(".png",  []),
                    (".jpg",  [cv2.IMWRITE_JPEG_QUALITY, 90]),
                    (".webp", [cv2.IMWRITE_WEBP_QUALITY, 90])]:
    ok, buf = cv2.imencode(ext, img, params)
    print(f"{ext:>6}: {len(buf):>8} bytes")

For vision work, the format decision tree is short. Ground truth and anything measured: lossless (PNG, or 16-bit TIFF for depth and scientific data), per the key insight above. Training photographs: whatever they already are; recompressing helps storage but compounds generation loss, and JPEG quality below about 75 measurably dents downstream accuracy, which is why JPEG-artifact augmentation appears in the robustness recipes of Chapter 21. Serving and archives: WebP or AVIF at high quality. And whenever throughput matters, remember from Chapter 0 that decode time differs across formats as much as size does; PNG's DEFLATE is often the training-loader bottleneck that JPEG's hardware-accelerated decoders are not.

This closes the chapter's arc: photons to charge, charge to samples, samples to levels, levels to color coordinates, and coordinates to bytes on disk. You now know what an image file really contains and what it has already lost. The next chapter starts changing images on purpose: per-pixel transformations, histograms, and thresholds, the simplest and most-used tools in the kit.