ConvNeXt: The CNN, Modernized

"They said attention had replaced me. So I borrowed its haircut, its diet, and its workout routine, kept my own bones, and showed up to the same benchmark looking exactly as good. Turns out it was never about the attention."

A Convolutional Network That Read the Transformer's Diary

Section 20.4 ended the efficiency branch of the story. This section returns to the accuracy frontier, where, in 2020 and 2021, the vision transformer (the subject of Chapter 22) and its hierarchical cousin the Swin transformer began topping ImageNet, and a narrative formed that convolution was obsolete. ConvNeXt, from Liu and colleagues at Facebook AI Research in 2022, is the rebuttal, and it is included here not as the "newest CNN" but as a model of how to think about architecture: change one thing, measure, keep what helps, and never confuse a bundled improvement with its cause.

1. The Transformer Scare Intermediate

For a few years around 2020 it genuinely looked as if convolution, the operation this entire chapter has been refining, was about to be retired. A vision transformer cuts an image into patches, embeds each patch as a token, and processes the tokens with self-attention (Chapter 22). With enough data and the right recipe it matched or beat the best CNNs, and because attention is a genuinely different mixing operation than convolution, the community reasonably attributed the gains to attention. The stakes were real: if the mixing operation was the cause, every CNN you just learned was a dead end. Hold that worry, because the next subsection dismantles it.

But transformers also arrived with a package of other changes: longer training schedules, stronger augmentation, the AdamW optimizer, LayerNorm instead of BatchNorm, GELU instead of ReLU, and a coarse "patchify" first layer. (GELU, the Gaussian Error Linear Unit, is a smooth alternative to the ReLU of Section 20.1: instead of the hard cutoff $\max(0, x)$, it scales each input by the probability that a standard Gaussian falls below it, so the curve bends gently through zero rather than kinking.) The confound was that no one had isolated which factor mattered. ConvNeXt's contribution is the isolation experiment.

2. Modernizing a ResNet, One Change at a Time Advanced

ConvNeXt starts from a standard ResNet-50 and applies the transformer-era changes incrementally, measuring ImageNet accuracy after each, in the spirit of a careful ablation. The changes fall into a few groups, sketched in Figure 20.5.1: a modern recipe (more epochs, better augmentation, AdamW), a macro redesign (a patchify stem and adjusted stage compute ratios; the patchify stem replaces ResNet's $7 \times 7$ stride-2 convolution plus max-pool with a single non-overlapping $4 \times 4$ stride-4 convolution, so each output position summarizes one disjoint $4 \times 4$ patch with no overlap, exactly the patch-embedding first step a vision transformer uses in Chapter 22), the ResNeXt trick of grouped or depthwise convolutions (ResNeXt is a ResNet variant that replaces each bottleneck's dense $3 \times 3$ with grouped convolution, the grouping idea of Section 20.4), an inverted bottleneck like MobileNetV2's, a large $7 \times 7$ depthwise kernel, and micro changes (GELU for ReLU, LayerNorm for BatchNorm, fewer activations and norms per block).

The single most striking number in the figure is the first jump: simply training the unchanged ResNet-50 with the modern recipe lifts accuracy by nearly three points (from about 76.1% to 78.8%), before any architectural change at all. The rest of the climb comes from macro and micro design, none of it attention. The destination, around 82% top-1 for the tiny variant, matches the Swin transformer it was chasing. The conclusion is precise: a pure convolutional network, given the same recipe and comparable macro design, is competitive with the transformer, so the mechanism was not the deciding factor.

The code below assembles those modernization steps into a single ConvNeXt block, so you can see exactly which parts it keeps from ResNet and which it borrows from the transformer.

# A ConvNeXt block keeps the ResNet residual and the depthwise convolution but
# borrows the transformer's habits: a large 7x7 kernel, LayerNorm, GELU, an
# inverted bottleneck, and only one activation and one norm per block.
import torch
import torch.nn as nn

class ConvNeXtBlock(nn.Module):
    """A ConvNeXt block: 7x7 depthwise -> LayerNorm -> 1x1 expand -> GELU -> 1x1."""
    def __init__(self, dim, expand=4, layer_scale=1e-6):
        super().__init__()
        self.dwconv = nn.Conv2d(dim, dim, 7, padding=3, groups=dim)  # large depthwise
        self.norm = nn.LayerNorm(dim, eps=1e-6)                      # channels-last LN
        self.pw1 = nn.Linear(dim, expand * dim)                     # 1x1 as Linear
        self.act = nn.GELU()                                        # GELU, not ReLU
        self.pw2 = nn.Linear(expand * dim, dim)
        # learnable per-channel scale on the residual branch, stabilizes training
        self.gamma = nn.Parameter(layer_scale * torch.ones(dim))

    def forward(self, x):                       # x: (B, C, H, W)
        skip = x
        x = self.dwconv(x)
        x = x.permute(0, 2, 3, 1)               # to (B, H, W, C) for LayerNorm/Linear
        x = self.pw2(self.act(self.pw1(self.norm(x))))
        x = self.gamma * x
        x = x.permute(0, 3, 1, 2)               # back to (B, C, H, W)
        return skip + x                          # the same residual addition as ResNet

blk = ConvNeXtBlock(96)
print(blk(torch.randn(1, 96, 56, 56)).shape)

Three design choices in that block are worth naming. The depthwise kernel is a large $7 \times 7$, echoing the large-kernel revival of RepLKNet and giving each block a wide receptive field in one step, the opposite of VGG's many-small-kernels philosophy from Section 20.2. The block uses only one activation and one normalization, far fewer than a ResNet block, matching the transformer's sparse use of nonlinearities. And the normalization is LayerNorm, computed over channels per spatial location, rather than BatchNorm; this removes the batch-size dependence that BatchNorm (from Chapter 19) imposes and matches transformer practice.

3. What ConvNeXt Tells You About Architecture Intermediate

ConvNeXt reframes a decade of this chapter. The architectures from LeNet to ResNet were genuine mechanism changes that each removed a real bottleneck. But by the 2020s the marginal architecture mattered less than the recipe and the data, a point Chapter 21 develops in full. Convolution and attention turn out to be two reasonable choices of mixing operation, and for many tasks the choice is less important than how you train. This does not make architecture irrelevant; it means architecture and recipe are entangled, and the honest comparison holds the recipe fixed. ConvNeXt V2 later pushed the design further by co-designing it with masked-autoencoder self-supervised pretraining, the subject of Chapter 25, showing the CNN can also benefit from the pretraining strategies that powered foundation models.

# Load ConvNeXt pretrained, re-headed for 10 classes, in one timm call. The V2
# variant pairs the same architecture with the masked-autoencoder pretraining of
# Chapter 25, so the block above is for understanding, not for production typing.
import timm
# ConvNeXt-Tiny, ImageNet-pretrained, matched to a Swin-Tiny in accuracy:
model = timm.create_model("convnext_tiny", pretrained=True, num_classes=10)
# ConvNeXt V2 with the masked-autoencoder pretraining of Chapter 25:
model_v2 = timm.create_model("convnextv2_tiny", pretrained=True)