"They handed me a blurry, noisy, half-missing photograph and asked for the truth. I gave them the most probable truth, which is the only kind I stock."
A Bayesian Restoration Filter
Chapter Overview
Every photograph you have ever taken was damaged before you saw it. Photons arrived at the sensor randomly, so every pixel carries shot noise. The lens spread each point of light into a small smear, and if your hand moved, the smear grew a tail. The scene's brightness range exceeded what the sensor could record, so the highlights clipped or the shadows drowned. And somewhere between sensor and screen, compression quietly threw information away. Image restoration is the discipline of undoing this damage, and image enhancement is its pragmatic cousin: making the picture look better even when "the original" is not strictly recoverable.
What makes restoration intellectually special is that it is the book's first true inverse problem. Everything in Chapter 2 through Chapter 6 ran forward: take an image, apply an operation, get a result. Restoration runs the movie backward. We observe a degraded image $g$, we model how it was produced from an ideal image $f$ (typically blur plus noise, $g = h \ast f + n$), and we try to invert the model. The catch, and the recurring theme of this chapter, is that the inversion is ill-posed: many different clean images explain the same dirty one, and naive inversion amplifies noise catastrophically. Every method here resolves the ambiguity the same way, by adding a prior: an assumption about what natural images look like. Smoothness, self-similarity, sparse gradients, each classical algorithm is a hand-written belief about images.
The journey runs from damage to remedy. Section 7.1 builds the vocabulary of degradation itself: where noise comes from physically, why its statistics are Poisson at heart, and how realistic degradation pipelines are assembled. Section 7.2 attacks noise, climbing from the Gaussian blur of Chapter 3 to the patch-based elegance of non-local means and BM3D. Section 7.3 takes on blur with frequency-domain deconvolution, where the Fourier machinery of Chapter 4 earns its keep: the Wiener filter and Richardson-Lucy iteration, including the algorithm that rescued the Hubble Space Telescope. Section 7.4 fills holes: inpainting by smoothness, by fast marching, and by copying texture. Section 7.5 asks how much resolution can honestly be recovered, from multi-frame fusion to the eve of deep super-resolution. Section 7.6 closes with dynamic range: merging bracketed exposures into high dynamic range radiance and tone mapping the result back onto an eight-bit screen.
This chapter is also a load-bearing wall for the rest of the book. The noise models of Section 7.1 return as the noise schedules of diffusion models in Chapter 33; denoising itself becomes the training objective of denoising autoencoders in Chapter 31 and, astonishingly, the generative engine of diffusion. Inpainting reappears as generative editing in Chapter 35, and super-resolution returns both at the edge in Chapter 28 and inside text-to-image upscalers in Chapter 34. When deep models eventually beat every method in this chapter, they did it by learning the priors we are about to write by hand. Knowing the hand-written versions is what lets you understand, debug, and distrust the learned ones.
Restoration is inference, not filtering: you model how the damage happened, you state what clean images look like, and the algorithm follows from those two commitments. Every method in this chapter, from the Wiener filter to exemplar inpainting, is the same Bayesian sentence with different nouns: find the image that best explains the observation while remaining plausible. Deep restoration networks did not change the sentence; they replaced the hand-written plausibility term with a learned one.
Learning Objectives
- Write down the degradation model $g = h \ast f + n$, identify the physical source of each term, and synthesize realistic noise (Gaussian, Poisson, salt-and-pepper, speckle) and full degradation pipelines in code.
- Denoise images with Gaussian, median, bilateral, non-local means, and BM3D filtering, and explain the prior each method encodes and where it fails.
- Deblur images by inverse filtering, the Wiener filter, and Richardson-Lucy iteration, and explain why regularization is unavoidable.
- Fill missing regions with PDE-based, fast-marching, and exemplar-based inpainting, and choose the right method from hole geometry.
- Reconstruct higher-resolution images from multiple aliased frames and explain why single-image interpolation cannot add information.
- Merge bracketed exposures into HDR radiance maps, tone map them for display, and decide when exposure fusion should replace the radiometric pipeline entirely.
Prerequisites
This chapter leans on most of Part I. From Chapter 1: Digital Image Fundamentals you need the sensor pipeline (where photons become numbers) and the PSNR and SSIM quality metrics, which we use to score every restoration. Chapter 3: Spatial Filtering & Convolution supplies convolution itself, plus the Gaussian, median, and bilateral filters that Section 7.2 builds upon. Chapter 4: The Frequency Domain & Multi-Scale Analysis is essential for Section 7.3 (deconvolution lives in the Fourier domain) and useful for Sections 7.5 and 7.6 (aliasing and pyramid blending). Chapter 5: Geometric Transformations & Image Warping provides interpolation and registration, which multi-frame super-resolution and HDR alignment both require. Chapter 2: Point Operations, Histograms & Thresholding helps with tone mapping, and Chapter 6: Morphology, Binary Images & Shape with preparing inpainting masks. If your environment needs setting up, start at Chapter 0.
Chapter Roadmap
- 7.1 Noise Models & Degradation Pipelines Where damage comes from: shot noise, read noise, blur, and compression, and how to model, synthesize, and measure each one before attempting any cure.
- 7.2 Classical Denoising: From Gaussian to Non-Local Means Denoising as the art of choosing what to average: local filters, edge-aware filters, patch self-similarity, and BM3D, the decade-long classical champion.
- 7.3 Deblurring & Deconvolution: Wiener & Richardson-Lucy Undoing convolution in the Fourier domain: why naive inversion explodes, how Wiener regularizes it, and the iterative method that fixed Hubble's eyesight.
- 7.4 Inpainting: Filling the Holes Reconstructing missing pixels by continuing structure, marching from the boundary, or copying texture, and knowing which trick the hole demands.
- 7.5 Classical Super-Resolution Why bicubic upscaling is not super-resolution, how sub-pixel shifts across frames buy genuine detail, and the back-projection loop that fuses them.
- 7.6 HDR Imaging & Tone Mapping Capturing more light range than the sensor allows and squeezing it onto a display: radiance recovery, tone-mapping operators, and exposure fusion.
The most famous deblurring job in history was performed on a telescope, not by one. When the Hubble Space Telescope launched in 1990 with a mirror ground 2.2 micrometers too flat, astronomers spent three years deconvolving its images with the Richardson-Lucy algorithm (Section 7.3) while waiting for the repair mission. The algorithm dates from 1972, the telescope from 1990, and the combination produced publishable science from a broken billion-dollar instrument. Software patience beat hardware perfection.
What's Next?
This chapter completes the conceptual core of Part I: you can now describe, filter, transform, measure, and repair an image. Chapter 8: Tools of the Trade: The Image Processing Stack steps back from algorithms to engineering: how OpenCV, scikit-image, Pillow, and friends divide the territory, how to choose among them, and how to assemble the techniques of Chapters 0 through 7 into pipelines that survive production. Every restoration method you just learned has a battle-tested library implementation; Chapter 8 shows you where each one lives and what it costs.
Bibliography & Further Reading
Foundational Papers
The paper that redefined a pixel's "neighborhood" as every similar patch in the image. Section 7.2's centerpiece, and the conceptual ancestor of attention mechanisms.
BM3D: group similar patches, denoise them jointly in a transform domain. The classical state of the art for a full decade and still the baseline deep denoisers must beat.
The two independent derivations of the Richardson-Lucy deconvolution of Section 7.3: a multiplicative update that respects the Poisson statistics of photon counting. Astronomers still run it nightly, half a century later.
Brought the word "inpainting" from art conservation into vision, framing hole filling as a PDE that continues isophotes into the missing region (Section 7.4).
The fast, simple inpainting algorithm behind OpenCV's cv2.INPAINT_TELEA flag: fill the hole inward from the boundary, in the order fast marching dictates.
Inpainting by copying patches, with a priority rule that lets structure propagate before texture fills in. The classical method that handles large holes credibly.
How to recover a camera's response curve and true scene radiance from a handful of bracketed exposures. The founding paper of Section 7.6's HDR pipeline.
Skip radiance recovery entirely: blend the best-exposed pixels from each bracket with a Laplacian pyramid. The pragmatic algorithm inside most phone "HDR" modes.
SRCNN: the three-layer network that ended the classical era of Section 7.5 and opened the deep one. Reads as a direct translation of sparse-coding super-resolution into convolutions.
Books
Chapter 5 ("Image Restoration and Reconstruction") is the canonical textbook treatment of noise models, inverse filtering, and Wiener theory, with worked frequency-domain examples.
Sections on image processing, computational photography, and HDR cover this chapter's territory with modern references; free PDF from the author.
Tools & Libraries
photo). Official API referenceHome of fastNlMeansDenoising, inpaint, the HDR calibration and merge classes, and every tone-mapping operator used in this chapter.
skimage.restoration API. Official documentationClean NumPy implementations of Wiener and Richardson-Lucy deconvolution, non-local means, biharmonic inpainting, and noise estimation, ideal for studying the algorithms.
Recent Research (2024-2026)
Restoration in the diffusion era: a generative prior so strong it reinvents missing content. The direct descendant of every prior in this chapter, with the same hallucination risks scaled up.
The degradation-pipeline insight of Section 7.1 weaponized: train a restorer entirely on synthetically damaged images, and it generalizes to real damage. Actively maintained and widely deployed.