Video Transformers

"They asked me to let every patch talk to every other patch. Then they added time, and now every patch wants to talk to every other patch in every other moment. I have run the numbers. We are going to need a smaller guest list."

An Attention Matrix Contemplating Its Own Quadratic Growth

The previous section built action recognition out of convolutions; this one rebuilds it out of attention, exactly as Chapter 22 rebuilt image classification. You should hold that chapter's machinery in mind, because the video transformer reuses the transformer block, the class token, and the positional embedding almost unchanged. The genuinely new content is twofold: how to tokenize a clip, and how to make attention over the resulting sequence affordable. Both reduce to managing the time axis, the same theme that organized 3D convolutions and two-stream networks in Section 26.2.

1. Tubelet Embedding: Tokens in Spacetime Beginner

The Vision Transformer turned an image into tokens by cutting it into non-overlapping patches and linearly embedding each. A video transformer has two natural ways to do the same for a clip. The first, used by TimeSformer, embeds each frame into spatial patches independently, producing $T$ groups of $N_s$ patch tokens. The second, the tubelet embedding of ViViT, cuts the clip into 3D blocks (tubelets) of size $t \times p \times p$ that span several frames at once, so each token already fuses a little motion before any attention happens. Tubelet embedding is the exact 3D analogue of patch embedding, and just as the 2D patch embedding turned out to be a strided 2D convolution in Chapter 22, the tubelet embedding is a strided 3D convolution, the same Conv3d primitive from Section 26.2.

The token count is what to watch. For a clip of $T$ frames at $H \times W$ resolution with patch size $p$ and temporal tubelet length $t$, the number of tokens is

For a typical $T = 32$, $H = W = 224$, $p = 16$, $t = 2$, that is $16 \times 14 \times 14 = 3136$ tokens, against the $196$ of a single ViT image. Since attention cost grows as $N^2$, the clip is roughly $(3136 / 196)^2 \approx 256$ times more expensive per attention layer than the image. Figure 26.3.1 shows the tubelet construction and the token-count growth.

The code below implements tubelet embedding as a single Conv3d whose kernel and stride both equal the tubelet size, the cleanest way to express it, exactly mirroring the patch-embedding-as-convolution trick of Chapter 22.

import torch
import torch.nn as nn

class TubeletEmbed(nn.Module):
    """Cut a clip into t x p x p tubelets and embed each as one token."""
    def __init__(self, dim=768, patch=16, tubelet=2, in_ch=3):
        super().__init__()
        # kernel == stride == tubelet size: non-overlapping 3D blocks
        self.proj = nn.Conv3d(in_ch, dim,
                              kernel_size=(tubelet, patch, patch),
                              stride=(tubelet, patch, patch))

    def forward(self, x):                       # x: (N, C, T, H, W)
        z = self.proj(x)                        # (N, dim, T/t, H/p, W/p)
        N, D, t, h, w = z.shape
        tokens = z.flatten(2).transpose(1, 2)   # (N, t*h*w, dim)
        return tokens, (t, h, w)

embed = TubeletEmbed(dim=768, patch=16, tubelet=2)
tokens, grid = embed(torch.randn(1, 3, 32, 224, 224))
print("tokens:", tokens.shape, "grid (t,h,w):", grid)
# tokens: torch.Size([1, 3136, 768]) grid (t,h,w): (16, 14, 14)

2. The Quadratic Wall, Again Intermediate

You met the quadratic cost of attention in Chapter 22, where it bit a document-analysis team at high resolution. Video makes it worse along a new axis. Joint space-time attention, where every token attends to every other token across all positions and all frames, is the most expressive option and the most expensive: with $N$ tokens the attention matrix has $N^2$ entries, and at $N = 3136$ that is nearly ten million entries per head per layer, before any of the activations. For longer clips or higher resolution it is simply infeasible on commodity hardware.

The remedy is to factorize attention along its axes, the same move that R(2+1)D made for convolution. Instead of one attention over all $N$ tokens, apply spatial attention within each frame (tokens attend only to other tokens in the same frame) and then temporal attention across frames (tokens at the same spatial location attend across time). The cost drops from $O(N^2)$ to roughly $O(N_s^2 \cdot T + T^2 \cdot N_s)$, where $N_s$ is the spatial token count per frame and, in this cost formula, $T$ counts the temporal token groups (the $T/t = 16$ of the running example, not the 32 raw frames). The table below makes the saving concrete.

Table 26.3.1 shows divided attention costing roughly one-fifteenth of joint attention while still letting every token influence every other through the two-step path. TimeSformer's ablation found this divided scheme to be the sweet spot: nearly the accuracy of joint attention at a fraction of the cost. The implementation in subsection 3 builds it.

3. Divided Space-Time Attention Intermediate

A divided space-time block applies a temporal attention sub-layer and a spatial attention sub-layer in sequence, each wrapped in the residual-and-norm structure of the standard transformer block from Chapter 22. The trick is purely in the reshaping: before temporal attention, group tokens so that those sharing a spatial location across frames sit together; before spatial attention, group tokens so that those sharing a frame sit together. The multi-head attention itself is unchanged. The code reuses a standard attention module and only manipulates the token layout.

import torch
import torch.nn as nn

class DividedSpaceTimeBlock(nn.Module):
    """TimeSformer-style block: temporal attention, then spatial attention."""
    def __init__(self, dim=768, heads=12):
        super().__init__()
        self.norm_t = nn.LayerNorm(dim)
        self.attn_t = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.norm_s = nn.LayerNorm(dim)
        self.attn_s = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.norm_m = nn.LayerNorm(dim)
        self.mlp = nn.Sequential(nn.Linear(dim, 4 * dim), nn.GELU(), nn.Linear(4 * dim, dim))

    def forward(self, x, t, h, w):              # x: (N, t*h*w, dim)
        B, _, D = x.shape
        # --- temporal attention: tokens at the same (h,w) attend across time ---
        xt = x.view(B, t, h * w, D).permute(0, 2, 1, 3).reshape(B * h * w, t, D)
        a, _ = self.attn_t(self.norm_t(xt), self.norm_t(xt), self.norm_t(xt))
        xt = (xt + a).reshape(B, h * w, t, D).permute(0, 2, 1, 3).reshape(B, t * h * w, D)
        # --- spatial attention: tokens in the same frame attend to each other ---
        xs = xt.view(B, t, h * w, D).reshape(B * t, h * w, D)
        a, _ = self.attn_s(self.norm_s(xs), self.norm_s(xs), self.norm_s(xs))
        xs = (xs + a).reshape(B, t * h * w, D)
        # --- shared MLP sub-layer ---
        return xs + self.mlp(self.norm_m(xs))

block = DividedSpaceTimeBlock(dim=768, heads=12)
x = torch.randn(1, 16 * 14 * 14, 768)           # the 3136 tokens from subsection 1
print("block out:", block(x, t=16, h=14, w=14).shape)  # block out: torch.Size([1, 3136, 768])

# Classify a clip with a Kinetics-finetuned VideoMAE from the Hugging Face hub.
# The processor and id2label map remove the tokenizing and label-decoding boilerplate.
from transformers import VideoMAEImageProcessor, VideoMAEForVideoClassification
import torch

proc = VideoMAEImageProcessor.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics")
model = VideoMAEForVideoClassification.from_pretrained("MCG-NJU/videomae-base-finetuned-kinetics")

# video: a list of 16 RGB frames as numpy arrays or PIL images
inputs = proc(list(video_frames), return_tensors="pt")   # handles resize + normalize
with torch.no_grad():
    logits = model(**inputs).logits
print(model.config.id2label[logits.argmax(-1).item()])    # e.g. "playing guitar"

4. VideoMAE: Redundancy as a Pretraining Advantage Advanced

Video transformers are even hungrier for data than image ones, which collides with the fact that labeled video is scarce and expensive. The answer, as in Chapter 25, is self-supervised pretraining, and the masked autoencoder transfers to video beautifully. VideoMAE masks a large fraction of the tubelets, feeds only the visible ones to the encoder, and asks a lightweight decoder to reconstruct the missing pixels. The striking detail is the masking ratio: where the image MAE of Chapter 25 masked about 75 percent of patches, VideoMAE masks 90 percent or more, and it does so with tube masking (the same spatial locations are masked across all frames) to prevent the model from cheating by copying an unmasked neighboring frame.

This extreme ratio is not an arbitrary tuning choice; it is the direct consequence of the temporal redundancy you measured in Section 26.1. Because adjacent frames carry so little new information, the reconstruction task stays hard and informative even when ninety percent of the input is hidden. The tiny visible fraction also makes pretraining fast. The encoder runs its quadratic-cost attention over only the visible ten percent of tokens, so dropping ninety percent of them cuts the attention cost by roughly a hundredfold, which is what makes masked video pretraining affordable. The code below builds a tube mask and shows how few tokens the encoder actually processes.

import torch

def tube_mask(t, h, w, mask_ratio=0.9):
    """Mask the same spatial positions across all t temporal groups."""
    num_spatial = h * w
    num_keep = int(num_spatial * (1 - mask_ratio))
    keep_spatial = torch.randperm(num_spatial)[:num_keep]      # same for every frame
    # build a (t*h*w,) boolean keep-mask by repeating the spatial pattern over time
    keep = torch.zeros(t, num_spatial, dtype=torch.bool)
    keep[:, keep_spatial] = True
    return keep.flatten()                                       # (t*h*w,)

t, h, w = 8, 14, 14
keep = tube_mask(t, h, w, mask_ratio=0.9)
print("total tubelet tokens:", keep.numel())   # total tubelet tokens: 1568
print("tokens seen by encoder:", keep.sum().item())  # tokens seen by encoder: 152
print("fraction processed: {:.0%}".format(keep.float().mean().item()))  # fraction processed: 10%

VideoMAE and its successors (VideoMAE V2, and the video branch of the joint image-video foundation models) are the modern recipe for pretraining a video transformer without a labeled Kinetics-scale dataset, and they connect directly to the self-supervision arc of Chapter 25. The pretrained backbone is then fine-tuned for action recognition exactly as the R(2+1)D backbone was in Section 26.2. Convolutions and transformers alike have now given us a clip-level label, but neither tells us how each individual pixel moved between two frames; Section 26.4 turns from recognizing motion to measuring it densely, rebuilding the optical flow of Chapter 15 in the deep era with RAFT.