From Frames to Clips: The Temporal Dimension

"Thirty frames a second, and twenty-nine of them are nearly identical to the one before. I used to find that insulting. Then I realized it was the only reason I could afford to watch at all."

A Video Decoder Making Peace With Redundancy

In the previous chapter you finished assembling the Vision Transformer and, with it, the full toolkit of single-image deep vision: convolutions, attention, detection, segmentation, and self-supervised backbones. Every one of those models took a tensor of shape (channels, height, width) and produced a label, a box, or a mask. Now we add time. The change is conceptually small and practically enormous, and this section is about meeting the practical part head on before the architectures of Section 26.2 assume it is solved. We will look at the shape of video data, the redundancy that defines its economics, the sampling choices that every video system makes whether or not its authors admit to them, and the tooling that decodes a clip into the tensor a network expects. The single question underneath all of it, sketched in the illustration below, is where to spend a fixed budget between the space and time axes.

1. The Shape of a Video Tensor Beginner

An image, as you have used it since Chapter 1, is a tensor of shape $(C, H, W)$: channels, height, width. A video is the same thing repeated over time, so it gains a temporal axis $T$ and becomes a 4D tensor $(C, T, H, W)$, or with a batch dimension the 5D tensor $(N, C, T, H, W)$ that PyTorch video models expect. The ordering is a convention worth committing to memory, because half the bugs in video code are an axis in the wrong place. TorchVision puts time after channels, $(N, C, T, H, W)$; some libraries and the raw output of decoders put it before, $(N, T, C, H, W)$, the frame-stack ordering. A single permute separates a working pipeline from a silent disaster.

The numbers attached to those axes are what make video expensive. A modest clip of $T = 16$ frames at $224 \times 224$ resolution with $3$ colour channels is $16 \times 3 \times 224 \times 224 \approx 2.4$ million values, sixteen times a single image. A full ten-second clip at the source frame rate of 30 frames per second is 300 frames, and the storage and compute scale linearly with $T$. This is why video models almost never see the full stream; they see a sampled clip, and the sampling is a design decision we return to in subsection 3. Figure 26.1.1 lays out the tensor and its axes.

The code below builds a synthetic clip tensor and demonstrates the two common axis orderings and the permute that converts between them. Run it and read the printed shapes against the figure to fix the convention in your mind.

import torch

# A batch of 2 clips, 16 frames each, RGB, 224x224.
N, C, T, H, W = 2, 3, 16, 224, 224

# Decoders usually hand you frame-stack order first: (N, T, C, H, W).
clip_frame_stack = torch.rand(N, T, C, H, W)
print("frame-stack order:", clip_frame_stack.shape)  # torch.Size([2, 16, 3, 224, 224])

# TorchVision video models want channel-then-time: (N, C, T, H, W).
clip_cttw = clip_frame_stack.permute(0, 2, 1, 3, 4).contiguous()
print("model-ready order:", clip_cttw.shape)          # torch.Size([2, 3, 16, 224, 224])

# How much memory does one clip occupy as float32?
bytes_per_clip = C * T * H * W * 4
print(f"one clip = {bytes_per_clip / 1e6:.1f} MB float32")  # one clip = 9.6 MB float32

2. Redundancy: The Tax and the Gift Beginner

The defining property of natural video is temporal redundancy. At 30 frames per second, consecutive frames are almost identical; the camera and the world both move slowly relative to the frame interval, so frame $t+1$ is mostly a small perturbation of frame $t$. This is why video compresses so well: the H.264 and H.265 codecs store a few full keyframes and encode everything in between as motion-compensated differences, which is also why decoding is not free, as we will see in subsection 4. The redundancy is a tax, because most of the frames you decode carry little new information, and it is simultaneously a gift, because it means you can throw most of them away and lose almost nothing. The illustration below captures both sides of that bargain at once.

We can quantify the redundancy directly. The mean absolute difference between adjacent frames is small and the difference between distant frames is large; the signal grows roughly with temporal separation. The code below measures this on a synthetic moving pattern, but the same trend holds for any real clip and is the empirical justification for the sparse sampling of subsection 3.

import torch

# Synthesize a clip where a bright square drifts across a dark canvas:
# slow drift means high redundancy between nearby frames.
T, H, W = 64, 64, 64
clip = torch.zeros(T, H, W)
for t in range(T):
    cx = 8 + t // 2          # square moves 1 pixel every 2 frames
    clip[t, 20:36, cx:cx + 16] = 1.0

# Mean absolute difference as a function of temporal gap.
for gap in (1, 2, 8, 32):
    diff = (clip[gap:] - clip[:-gap]).abs().mean().item()
    print(f"gap={gap:2d} frames -> mean abs diff = {diff:.4f}")

# gap= 1 frames -> mean abs diff = 0.0039
# gap= 2 frames -> mean abs diff = 0.0078
# gap= 8 frames -> mean abs diff = 0.0307
# gap=32 frames -> mean abs diff = 0.1147

This measurement explains a recurring theme: video is forgiving of frame loss in a way that audio or a single image is not. Drop every other frame and a human, or a network, still reads the action perfectly. That tolerance is what makes the aggressive sampling and the extreme masking ratios of Section 26.3 possible. It is also the reason the two-stream networks of Section 26.2 can afford to compute expensive optical flow on only a handful of frames: the motion field changes slowly enough that a sparse set of flow estimates captures most of the dynamics.

3. Frame Sampling Strategies Intermediate

Given a long video and a fixed budget of $T$ frames, how do you choose which frames to keep? This is the sampling problem, and the three dominant answers each encode a different assumption about where the relevant motion lives. Dense sampling takes $T$ consecutive frames from a short window, capturing fine-grained motion but only a fraction of a second of context, which suits brief actions like a golf swing. Uniform sampling spreads $T$ frames evenly across the whole video, covering the full duration at the cost of temporal resolution, which suits long actions whose phases unfold over seconds. Segment-based sampling, introduced by the Temporal Segment Network, splits the video into $T$ equal segments and draws one random frame from each; this combines full coverage with the stochasticity that acts as data augmentation during training.

Figure 26.1.2 contrasts the three strategies on a single 300-frame strip, and the contrast is the whole point: dense sampling clusters its frames into one short window, uniform sampling spreads them evenly across the entire clip, and segment-based sampling spreads them evenly too but jitters each pick inside its own segment. The choice interacts with the architecture. A SlowFast network (Section 26.2) deliberately samples one pathway densely and the other sparsely, so it needs both. A video transformer with a fixed token budget almost always samples uniformly. The implementation below produces all three index sets so you can see how few frames each keeps from a 300-frame clip.

import torch

def dense_indices(num_frames, clip_len, stride=1, start=0):
    """T consecutive frames (with optional stride) from one window."""
    idx = torch.arange(start, start + clip_len * stride, stride)
    return idx.clamp(max=num_frames - 1)

def uniform_indices(num_frames, clip_len):
    """T frames spread evenly across the whole video."""
    return torch.linspace(0, num_frames - 1, clip_len).round().long()

def segment_indices(num_frames, clip_len, train=True):
    """Temporal Segment Network: one frame per equal segment."""
    bounds = torch.linspace(0, num_frames, clip_len + 1).long()
    out = []
    for i in range(clip_len):
        lo, hi = bounds[i].item(), max(bounds[i + 1].item(), bounds[i].item() + 1)
        # random frame within the segment during training, center at test time
        pick = torch.randint(lo, hi, (1,)).item() if train else (lo + hi) // 2
        out.append(pick)
    return torch.tensor(out).clamp(max=num_frames - 1)

NUM, CLIP = 300, 8
print("dense  :", dense_indices(NUM, CLIP, stride=2).tolist())
print("uniform:", uniform_indices(NUM, CLIP).tolist())
print("segment:", segment_indices(NUM, CLIP, train=True).tolist())

# dense  : [0, 2, 4, 6, 8, 10, 12, 14]
# uniform: [0, 43, 86, 128, 171, 214, 257, 299]
# segment: [22, 49, 99, 141, 169, 226, 251, 284]

4. Decoding Clips Into Tensors Intermediate

The last practical hurdle is decoding: turning a compressed video file into the frame tensors the sampler indexes. Because codecs store most frames as differences from earlier ones, you cannot in general jump to an arbitrary frame without decoding from the nearest keyframe forward, which makes random frame access surprisingly expensive. TorchVision exposes two interfaces. The simple read_video reads an entire clip (or a time range) into memory at once; the streaming VideoReader seeks and yields frames one at a time, which is far more memory-efficient for long videos where you only want a handful of frames. The code below uses the streaming reader to apply uniform sampling without ever holding the whole video in memory.

import torch
from torchvision.io import VideoReader

def load_clip(path, clip_len=16):
    """Uniformly sample clip_len frames from a video into a (C, T, H, W) tensor."""
    reader = VideoReader(path, "video")
    meta = reader.get_metadata()["video"]
    duration = meta["duration"][0]          # seconds
    fps = meta["fps"][0]
    num_frames = int(duration * fps)

    # timestamps (in seconds) of the frames we want
    targets = torch.linspace(0, duration * (1 - 1e-3), clip_len).tolist()

    frames = []
    for t in targets:
        reader.seek(t)                      # jump to nearest decodable frame
        frame = next(reader)["data"]        # uint8 tensor (C, H, W)
        frames.append(frame)

    clip = torch.stack(frames, dim=1)       # (C, T, H, W)
    return clip.float() / 255.0             # scale to [0, 1]

# clip = load_clip("game.mp4", clip_len=16)
# print(clip.shape)  ->  torch.Size([3, 16, 224, 224])  (after a resize transform)

In a real training loop you would wrap this in a Dataset, add a spatial resize and the per-channel normalization from Chapter 21, and let the DataLoader batch clips into the 5D tensor of subsection 1. The normalization statistics are the same idea you first met as the histogram-based intensity statistics of Chapter 2, now computed once over a video dataset and applied per channel. The one new wrinkle is that augmentations must be applied consistently across all frames of a clip; a random crop or flip chosen per frame would destroy the temporal coherence the model is trying to learn.

from pytorchvideo.data import labeled_video_dataset, make_clip_sampler
from pytorchvideo.transforms import ApplyTransformToKey, UniformTemporalSubsample
from torchvision.transforms import Compose, Lambda, Resize

transform = ApplyTransformToKey(
    key="video",
    transform=Compose([
        # Subsample to 16 frames, scale to [0,1], and resize, all clip-consistent.
        UniformTemporalSubsample(16),       # 16 frames per clip
        Lambda(lambda x: x / 255.0),
        Resize((224, 224)),
    ]),
)
dataset = labeled_video_dataset(
    data_path="kinetics/train.csv",
    clip_sampler=make_clip_sampler("random", clip_duration=2.0),  # 2-second clips
    transform=transform,
    decode_audio=False,
)
# next(iter(dataset))["video"].shape  ->  (3, 16, 224, 224)