Choosing an Architecture in Practice

"The leaderboard crowns the network with the most accuracy. The pager at 3 a.m. crowns the network that fit in memory and answered in time. Choose the one that will not wake you up."

A Backbone That Has Been On Call

You now know the great architectures and, more usefully, the bottleneck each removed. Section 20.5 left you with the entangled-recipe lesson: architecture and training are inseparable, and the modern workflow rarely starts from a blank network. This closing section is deliberately practical. It assumes you have a task, a hardware target, and a deadline, and it shows how to pick a backbone without re-deriving the field.

1. The Four Numbers on Every Stats Sheet Beginner

Every architecture is summarized by four quantities, and confusing them is the most common selection error. Parameters count the weights; they determine model file size and a rough lower bound on memory, but not speed. FLOPs (floating-point operations, usually reported as multiply-adds per image) estimate compute, but as Section 20.4 warned, they correlate only loosely with wall-clock time because memory bandwidth, not arithmetic, often dominates. Latency is the measured time for one forward pass on your target hardware, the number that actually matters for a real-time system, and it must be measured, not predicted. Accuracy (top-1 on ImageNet, or your task's metric) is the headline, but it is meaningless without the cost it was bought at. Table 20.6.1 grounds these with representative figures.

Read the table as a frontier, not a ranking. MobileNetV3-Small uses on the order of a couple hundred times fewer FLOPs than ConvNeXt-Base for about sixteen fewer accuracy points, an excellent trade on a phone and a terrible one on a server with idle GPUs. The ResNet-50 row is bolded in your mind for a reason: it remains the field's reference point, well understood, widely supported, and with the modern weights of Section 20.3 still competitive. Figure 20.6.1 plots the frontier these rows trace.

2. Picking a Backbone with timm Beginner

In practice you do not implement any of this chapter's networks; you select one from a library and load pretrained weights. The timm library is the standard catalogue, with hundreds of backbones, benchmarked accuracy and throughput tables, and a one-line model factory. The workflow is: filter the catalogue to candidates that fit your cost budget, load each pretrained, measure latency on your hardware, and pick the cheapest that clears your accuracy target after fine-tuning.

# Benchmark candidate backbones on YOUR hardware, the number that actually
# decides a deployment. Warm-up passes absorb one-time compile and allocation
# costs so the timed loop reports steady-state latency, not first-call overhead.
import timm, torch, time

# 1. List candidate backbones by name pattern:
candidates = ["mobilenetv3_large_100", "efficientnet_b0", "resnet50", "convnext_tiny"]

# 2. Load each pretrained and measure latency on YOUR hardware (the real metric):
device = "cuda" if torch.cuda.is_available() else "cpu"
x = torch.randn(1, 3, 224, 224, device=device)
for name in candidates:
    model = timm.create_model(name, pretrained=True).eval().to(device)
    with torch.no_grad():
        for _ in range(5):           # warm up (kernels compile, caches fill)
            model(x)
        t0 = time.perf_counter()
        for _ in range(50):
            model(x)
        if device == "cuda":
            torch.cuda.synchronize()
    n_params = sum(p.numel() for p in model.parameters()) / 1e6
    ms = (time.perf_counter() - t0) / 50 * 1000
    print(f"{name:24s} {n_params:6.1f}M params  {ms:6.2f} ms/image")

# Adapt a pretrained ImageNet backbone to your own label set in one argument,
# then freeze the trunk and unfreeze only the new head for linear probing. This
# is the most common production task, done without manual layer surgery.
import timm
# Load a pretrained backbone but replace the 1000-class head with your own:
model = timm.create_model("convnext_tiny", pretrained=True, num_classes=37)
# timm swaps the classifier, keeps the pretrained trunk, ready to fine-tune.
# Freeze the trunk for linear probing, or leave it trainable for full fine-tuning:
for p in model.parameters(): p.requires_grad = False
for p in model.get_classifier().parameters(): p.requires_grad = True

3. Transfer Learning Is the Default Intermediate

The most important practical fact in this section is that you should almost never train an architecture from scratch. A backbone pretrained on a large dataset has already learned the general visual features (the edge, texture, and part detectors you visualized in Chapter 19), and those features transfer to nearly any image task. Fine-tuning such a backbone on a few thousand of your own images routinely beats training the same architecture from random initialization on the same data, often by a wide margin, and trains in a fraction of the time. The transfer-learning arc of this book runs from here through the foundation backbones of Chapter 25 to the fine-tuned generators of Chapter 34; this is its first concrete payoff.

There are two transfer styles. Linear probing freezes the trunk and trains only a new head, fast and data-thrifty, best when your data is scarce or very similar to the pretraining domain. Full fine-tuning updates the whole network at a small learning rate, stronger when you have more data or a domain shift (medical scans, satellite imagery, microscopy). The decision and its details, learning-rate schedules, layer-wise rates, when to unfreeze, belong to Chapter 21, but the default posture is set here: start pretrained, fine-tune, only consider from-scratch if a strong reason demands it.

4. A Decision Checklist Beginner

Putting the chapter together, here is a procedure for choosing an architecture for a new vision project. It is ordered so the cheapest, highest-leverage decisions come first.

1. State the hard constraints first. Target hardware, maximum latency, memory budget, and the minimum acceptable accuracy. These prune the candidate set before you look at any model.
2. Default to a pretrained backbone. Begin with a ResNet-50 or ConvNeXt-Tiny on a server, an EfficientNet-B0 or MobileNetV3 on a device. Only deviate with a reason.
3. Match the backbone to the budget using the frontier. Pick the frontier model (Figure 20.6.1) whose accuracy clears your bar at the lowest cost your hardware allows; do not overshoot.
4. Re-head and fine-tune, do not train from scratch. Use linear probing if data is scarce or in-domain, full fine-tuning if data is ample or out-of-domain.
5. Measure latency on the real target, not FLOPs. Run the bake-off above on the actual chip; FLOPs can mislead by an order of magnitude.
6. Match the recipe before comparing models. Per Section 20.5, never compare two architectures trained with different recipes; hold the recipe fixed or your comparison is meaningless.
7. Only then consider exotic options. Vision transformers (Chapter 22), quantization and pruning (Chapter 28), or a custom design, when a strong, pretrained CNN provably cannot meet the constraints.

This checklist is exactly what the Hands-On Lab below turns into a runnable tool. You will profile the chapter's backbones on the four numbers, draw the frontier from your own measurements, and then fine-tune the model the checklist chooses, ending with a single decision report you could paste into a project ticket.

pip install torch torchvision timm matplotlib

import timm
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
NAMES = ["resnet50", "efficientnet_b0", "mobilenetv3_small_100", "convnext_tiny"]

# TODO: build a dict {name: model} of pretrained backbones in eval mode on `device`.
# Hint: timm.create_model(name, pretrained=True), then .eval().to(device)
models = {}
for name in NAMES:
    ...
print({n: type(m).__name__ for n, m in models.items()})

def param_count_m(model):
    # TODO: return the total parameter count in millions.
    # Hint: sum p.numel() for p in model.parameters(), divide by 1e6
    ...

# FLOPs via fvcore if available, else skip gracefully.
def gflops(model):
    try:
        from fvcore.nn import FlopCountAnalysis
        x = torch.randn(1, 3, 224, 224, device=next(model.parameters()).device)
        return FlopCountAnalysis(model, x).total() / 1e9
    except Exception:
        return float("nan")

for n, m in models.items():
    print(f"{n:24s} params={param_count_m(m):6.1f}M  GFLOPs={gflops(m):6.2f}")

import time

@torch.no_grad()
def latency_ms(model, runs=30, warmup=5):
    x = torch.randn(1, 3, 224, 224, device=device)
    for _ in range(warmup):
        model(x)
    if device == "cuda":
        torch.cuda.synchronize()
    times = []
    for _ in range(runs):
        t0 = time.perf_counter()
        model(x)
        if device == "cuda":
            torch.cuda.synchronize()
        # TODO: append the elapsed milliseconds for this run to `times`
        ...
    times.sort()
    return times[len(times) // 2]  # median

for n, m in models.items():
    print(f"{n:24s} {latency_ms(m):7.2f} ms on {device}")

import matplotlib.pyplot as plt

# Published ImageNet top-1 (%), from torchvision/timm model cards (Table 20.6.1).
ACC = {"resnet50": 80.9, "efficientnet_b0": 77.7,
       "mobilenetv3_small_100": 67.7, "convnext_tiny": 82.1}

lat = {n: latency_ms(m) for n, m in models.items()}

fig, ax = plt.subplots(figsize=(6, 4))
for n in models:
    # TODO: scatter point (lat[n], ACC[n]) and annotate it with the model name
    ...
ax.set_xlabel(f"Measured latency (ms, {device})"); ax.set_ylabel("ImageNet top-1 (%)")
ax.set_title("Accuracy vs measured latency"); plt.tight_layout(); plt.show()

def choose(acc, lat, min_acc, max_lat_ms):
    # Keep only models that clear BOTH constraints, then pick the fastest survivor.
    feasible = {n: lat[n] for n in acc if acc[n] >= min_acc and lat[n] <= max_lat_ms}
    # TODO: return the name with the smallest latency among `feasible`,
    #       or None if nothing qualifies. Hint: min(feasible, key=feasible.get)
    ...

pick = choose(ACC, lat, min_acc=75.0, max_lat_ms=max(lat.values()) + 1)
print("Checklist selects:", pick)

import torch.nn as nn
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.datasets import Flowers102

tf = transforms.Compose([transforms.Resize((224, 224)), transforms.ToTensor()])
train = Flowers102(root="./data", split="train", download=True, transform=tf)
loader = DataLoader(train, batch_size=32, shuffle=True, num_workers=0)

net = timm.create_model(pick, pretrained=True, num_classes=102).to(device)
# TODO: freeze every parameter, then unfreeze ONLY the classifier head
#       (timm exposes it via net.get_classifier()) so this is a linear probe.
...

opt = torch.optim.Adam((p for p in net.parameters() if p.requires_grad), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()
net.train()
for step, (xb, yb) in enumerate(loader):
    xb, yb = xb.to(device), yb.to(device)
    opt.zero_grad(); loss = loss_fn(net(xb), yb); loss.backward(); opt.step()
    if step % 10 == 0: print(f"step {step:3d}  loss {loss.item():.3f}")
    if step == 60: break

print("=== Backbone decision report ===")
print(f"Shortlist : {', '.join(models)}")
for n in models:
    print(f"  {n:24s} {param_count_m(models[n]):5.1f}M  "
          f"{lat[n]:6.2f} ms  {ACC[n]:.1f}% top-1")
# TODO: print the selected model, why it won (cheapest that cleared the bar),
#       and the constraints it was chosen under.
...

import time
import torch
import torch.nn as nn
import timm
import matplotlib.pyplot as plt
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.datasets import Flowers102

device = "cuda" if torch.cuda.is_available() else "cpu"
NAMES = ["resnet50", "efficientnet_b0", "mobilenetv3_small_100", "convnext_tiny"]
ACC = {"resnet50": 80.9, "efficientnet_b0": 77.7,
       "mobilenetv3_small_100": 67.7, "convnext_tiny": 82.1}

# Step 1: load pretrained backbones
models = {}
for name in NAMES:
    try:
        models[name] = timm.create_model(name, pretrained=True).eval().to(device)
    except Exception as e:
        print(f"skip {name}: {e}")

# Step 2: parameters and FLOPs
def param_count_m(model):
    return sum(p.numel() for p in model.parameters()) / 1e6

def gflops(model):
    try:
        from fvcore.nn import FlopCountAnalysis
        x = torch.randn(1, 3, 224, 224, device=next(model.parameters()).device)
        return FlopCountAnalysis(model, x).total() / 1e9
    except Exception:
        return float("nan")

# Step 3: latency
@torch.no_grad()
def latency_ms(model, runs=30, warmup=5):
    x = torch.randn(1, 3, 224, 224, device=device)
    for _ in range(warmup):
        model(x)
    if device == "cuda":
        torch.cuda.synchronize()
    times = []
    for _ in range(runs):
        t0 = time.perf_counter()
        model(x)
        if device == "cuda":
            torch.cuda.synchronize()
        times.append((time.perf_counter() - t0) * 1000)
    times.sort()
    return times[len(times) // 2]

lat = {n: latency_ms(m) for n, m in models.items()}

# Step 4: frontier plot
fig, ax = plt.subplots(figsize=(6, 4))
for n in models:
    ax.scatter(lat[n], ACC[n])
    ax.annotate(n, (lat[n], ACC[n]), fontsize=8, xytext=(4, 4),
                textcoords="offset points")
ax.set_xlabel(f"Measured latency (ms, {device})")
ax.set_ylabel("ImageNet top-1 (%)")
ax.set_title("Accuracy vs measured latency")
plt.tight_layout(); plt.savefig("frontier.png", dpi=120)

# Step 5: checklist selection
def choose(acc, lat, min_acc, max_lat_ms):
    feasible = {n: lat[n] for n in acc if acc[n] >= min_acc and lat[n] <= max_lat_ms}
    if not feasible:
        return None
    return min(feasible, key=feasible.get)

pick = choose(ACC, lat, min_acc=75.0, max_lat_ms=max(lat.values()) + 1)
print("Checklist selects:", pick)

# Step 6: re-head and linear-probe fine-tune on Flowers102
tf = transforms.Compose([transforms.Resize((224, 224)), transforms.ToTensor()])
train = Flowers102(root="./data", split="train", download=True, transform=tf)
loader = DataLoader(train, batch_size=32, shuffle=True, num_workers=0)

net = timm.create_model(pick, pretrained=True, num_classes=102).to(device)
for p in net.parameters():
    p.requires_grad = False
for p in net.get_classifier().parameters():
    p.requires_grad = True

opt = torch.optim.Adam((p for p in net.parameters() if p.requires_grad), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()
net.train()
for step, (xb, yb) in enumerate(loader):
    xb, yb = xb.to(device), yb.to(device)
    opt.zero_grad()
    loss = loss_fn(net(xb), yb)
    loss.backward()
    opt.step()
    if step % 10 == 0:
        print(f"step {step:3d}  loss {loss.item():.3f}")
    if step == 60:
        break

# Step 7: decision report
print("=== Backbone decision report ===")
print(f"Shortlist : {', '.join(models)}")
for n in models:
    print(f"  {n:24s} {param_count_m(models[n]):5.1f}M  "
          f"GFLOPs={gflops(models[n]):5.2f}  {lat[n]:6.2f} ms  {ACC[n]:.1f}% top-1")
print(f"Selected  : {pick}")
print("Why       : cheapest model that cleared the 75.0% accuracy floor within "
      "the latency budget; fine-tuned, not trained from scratch.")