Images as Arrays: Pixels, Channels & Dtypes

"People keep telling me I have hidden depth. Technically I have three channels, a dtype, and trust issues about my value range."

A Mildly Philosophical Pixel

This is the first section of the book, so we begin at the absolute beginning: not with light, not with cameras (that story is told in Chapter 1), but with the object your code actually touches. Open an interpreter and follow along; every snippet below runs as written, with no image files required. The illustration below captures the single idea this section turns on.

1. The Image Is the Array Beginner

A grayscale image is a two-dimensional grid of brightness values. In NumPy terms, it is a 2-D array of shape $(H, W)$: $H$ rows and $W$ columns. The element at row $y$, column $x$ is the pixel at that location, and mathematically we treat the image as a function

$$I : \{0, \dots, H-1\} \times \{0, \dots, W-1\} \rightarrow \mathbb{V},$$

where $\mathbb{V}$ is the set of representable values (for the common 8-bit case, the integers $0$ to $255$). Two conventions surprise newcomers immediately. First, the origin is the top-left corner, not the bottom-left as in mathematics class: row indices grow downward. Second, indexing is img[row, col], that is, vertical coordinate first. Figure 0.1.1 makes both conventions concrete, and Section 0.4 returns to the trouble they cause when mixed with the $(x, y)$ convention used by drawing and resizing functions.

Let us build exactly this kind of object from nothing. The code below synthesizes a horizontal brightness ramp, then interrogates it the way you should interrogate every image you ever load: shape, dtype, extreme values.

import numpy as np

# A horizontal ramp: each row runs 0..255 left to right.
row = np.linspace(0, 255, 320, dtype=np.uint8)   # one row, 320 values
img = np.tile(row, (240, 1))                     # stack it 240 times

print(img.shape)          # (240, 320)  -> 240 rows (H), 320 columns (W)
print(img.dtype)          # uint8
print(img.min(), img.max())  # 0 255
print(img[120, 160])      # 127  -> the pixel at row 120, column 160

Everything you know about NumPy now applies to pictures. Slicing crops: img[60:180, 80:240] is a rectangular region of interest. Fancy indexing selects: img[img > 200] = 255 brightens highlights, a one-line preview of the thresholding we study properly in Chapter 2. Aggregations measure: img.mean() is the average brightness of the whole frame. There is no separate "image API" to learn for any of this; the array API is the image API.

2. Channels: The Third Axis Beginner

Color enters as a third axis. A color image is an array of shape $(H, W, 3)$, where the last axis holds the three color components of each pixel. In the RGB convention used by Pillow, scikit-image, Matplotlib, and essentially all deep learning code, those components are red, green, and blue in that order; OpenCV famously stores them reversed, as BGR, a historical accident dissected in Section 0.4. Indexing img[y, x] now returns a length-3 vector rather than a scalar, and img[:, :, 0] peels off an entire channel as a 2-D array. Figure 0.1.2 shows the geometry: three aligned planes stacked along the last axis.

The following snippet constructs a tiny color image by direct channel assignment, then verifies what landed where. Building images by hand like this is more than a toy exercise: it is the standard way to create test fixtures for vision code, because you know the ground truth of every pixel.

import numpy as np

img = np.zeros((100, 300, 3), dtype=np.uint8)  # black canvas, RGB order
img[:, :100, 0] = 255      # left third: pure red
img[:, 100:200, 1] = 255   # middle third: pure green
img[:, 200:, 2] = 255      # right third: pure blue

print(img[50, 50])    # [255   0   0]  red pixel
print(img[50, 150])   # [  0 255   0]  green pixel
print(img[50, 250])   # [  0   0 255]  blue pixel

red_plane = img[:, :, 0]
print(red_plane.shape, red_plane.mean().round(1))  # (100, 300) 85.0

Two more channel layouts deserve a mention now so they do not ambush you later. Images with transparency carry a fourth alpha channel, shape $(H, W, 4)$. And deep learning frameworks prefer the channel axis first: a PyTorch image tensor is $(C, H, W)$, and a batch is $(N, C, H, W)$. The conversion is a single np.transpose(img, (2, 0, 1)), but forgetting it is a rite of passage we will formalize in Chapter 18.

3. Dtypes: The Contract About What Numbers Mean Intermediate

The dtype of an image array is not a storage detail; it is a contract. It declares how much memory each value occupies, what range it can represent, and, by strong convention, what range it is expected to occupy. The three dtypes you will meet constantly are summarized in Table 0.1.1.

An 8-bit channel offers $2^8 = 256$ distinct levels, which is roughly the limit of what human eyes distinguish under normal viewing; a 16-bit channel offers $2^{16} = 65536$ levels, which matters when you plan to stretch shadows or process medical scans, as discussed when we treat bit depth and dynamic range in Chapter 1. Floats exist not for storage but for mathematics: the moment you average, filter, or feed an image to a network, you want real-number arithmetic without overflow. Memory follows directly from the contract: a 12-megapixel RGB photo costs $4000 \times 3000 \times 3 = 36$ MB as uint8 and four times that, 144 MB, as float32. That factor of four decides batch sizes on GPUs for the rest of the book.

The danger zone is integer arithmetic. Unsigned 8-bit values wrap around modulo 256, so adding brightness can make pixels darker, as the meter in the illustration below shows:

import numpy as np

a = np.full((2, 2), 200, dtype=np.uint8)
b = np.full((2, 2), 100, dtype=np.uint8)

print(a + b)
# [[44 44]
#  [44 44]]      because (200 + 100) % 256 == 44: wraparound!

# The safe patterns:
print((a.astype(np.uint16) + b).clip(0, 255).astype(np.uint8))
# [[255 255]
#  [255 255]]    promote, clip, demote

mean = (a.astype(np.float32) + b.astype(np.float32)) / 2
print(mean.astype(np.uint8))
# [[150 150]
#  [150 150]]    averages need float (or uint16) intermediates

Formally, uint8 addition computes $(a + b) \bmod 256$, while what you almost always want is the saturating sum $\min(a + b, 255)$. OpenCV's cv2.add saturates for exactly this reason, one of several arithmetic conventions we contrast carefully in Section 0.4. The second classic dtype accident is conversion without rescaling: calling astype(np.uint8) on a float image in $[0, 1]$ truncates nearly everything to zero. Conversions must rescale, $x_{\text{uint8}} = \lfloor 255 \, x_{\text{float}} + 0.5 \rfloor$, not merely cast.

from skimage.util import img_as_float32, img_as_ubyte

f = img_as_float32(img_u8)   # uint8 0..255  -> float32 0.0..1.0
u = img_as_ubyte(f)          # float 0..1    -> uint8 0..255, rounded and clipped

4. Under the Hood: Memory, Strides, Views & Copies Advanced

One level beneath shape and dtype sits the machinery that makes NumPy fast: a flat block of bytes plus strides, the number of bytes to step to move one position along each axis. For a contiguous $(H, W, 3)$ uint8 image the strides are $(3W, 3, 1)$: one byte to the next channel, three bytes to the next pixel, $3W$ bytes to the next row. The byte offset of element $(y, x, c)$ is simply

$$\text{offset}(y, x, c) = y \cdot 3W + x \cdot 3 + c.$$

The three numbers multiplying $y$, $x$, and $c$ here are exactly the strides $(3W, 3, 1)$, which is no coincidence: a stride is just how far you jump in the flat buffer to move one step along an axis, so the offset of any element is each coordinate times its stride, added up. Figure 0.1.3 unrolls a small image into its flat buffer to make this concrete, and shows why a slice can read the same bytes without moving any of them.

Why should a practitioner care? Because strides explain the single most consequential performance fact about NumPy images: slicing does not copy. A crop like roi = img[100:200, 50:150] creates a view, a new array object pointing into the same bytes with adjusted offsets and strides. Views make cropping free, but they also mean that writing into a view writes into the original, and that some downstream consumers (certain OpenCV functions, serialization, C extensions) require contiguous memory and will either copy behind your back or refuse non-contiguous input.

import numpy as np

img = np.zeros((240, 320, 3), dtype=np.uint8)
print(img.strides)            # (960, 3, 1)   bytes per step along each axis

roi = img[100:200, 50:150]    # a view: no pixels are copied
roi[:] = 255                  # ... so this writes into img itself!
print(img[150, 100])          # [255 255 255]  the "original" changed

flipped = img[::-1]           # vertical flip as a negative-stride view
print(flipped.strides)        # (-960, 3, 1)
print(flipped.flags['C_CONTIGUOUS'])   # False

safe = img[100:200, 50:150].copy()     # an independent crop
print(np.shares_memory(img, roi), np.shares_memory(img, safe))  # True False

To feel how much "no copy" buys you, picture cropping the top-left quarter of that 12-megapixel photo a thousand times: a thousand views allocate three small array objects each and touch zero pixels, finishing in microseconds, while a thousand .copy() calls move nine megabytes apiece and grind through gigabytes of memory traffic. The slice that looks like it cut out a region actually cut out nothing; it just wrote down new directions for reading the same bytes. That is why img[::-1] can flip a frame instantly: a vertical flip is one minus sign on a stride, not a single pixel moved. The rule of thumb: treat views as read-only windows unless mutation of the parent is exactly what you intend, and reach for .copy() whenever an image crosses a function boundary that might write. We will see this rule earn its keep in Section 0.4, where an in-place ROI edit corrupts a source image, and again throughout Chapter 5, where geometric operations must decide between viewing and resampling.

5. Looking Ahead: From Arrays to Everything Else Beginner

Every later chapter consumes the model built here. Histograms in Chapter 2 are statistics over these array values. Convolution in Chapter 3 slides kernels across these axes. PyTorch tensors in Chapter 18 are this same object with gradients and a transposed channel axis, and the noise that diffusion models learn to remove in Chapter 33 is sampled into float arrays shaped exactly like the ones you built today. The next section widens the view from the object itself to the ecosystem of libraries that all agreed, with one famous color-order exception, to speak this array language.