Efficient Designs: MobileNet, ShuffleNet & EfficientNet

"Accuracy is easy when someone else pays the electricity bill. Run me on a phone in someone's pocket and suddenly every multiply-add is a moral decision."

A Network Learning to Live Within Its Means

ResNet (Section 20.3) made depth cheap to train but not cheap to run; a ResNet-50 needs about four billion multiply-adds per image, fine on a server, painful on a battery. From roughly 2017 onward the frontier of architecture research split: one branch chased ever-higher accuracy, the other asked how little compute could buy a usable model. This section follows the efficiency branch, and its ideas now sit inside almost every model that ships to a real device, the topic of Chapter 28.

1. Depthwise-Separable Convolution Intermediate

A standard convolution with kernel size $K$, $C_{in}$ input channels, and $C_{out}$ output channels does two jobs in one shot: it filters spatially (the $K \times K$ window) and it combines channels (summing across all $C_{in}$ inputs for each output). Its cost per output pixel is $K^2 \cdot C_{in} \cdot C_{out}$ multiply-adds. MobileNet's insight is that these two jobs can be separated. First a depthwise convolution applies one $K \times K$ filter to each input channel independently, doing only spatial work, at cost $K^2 \cdot C_{in}$. Then a pointwise convolution (a $1 \times 1$ convolution, the channel mixer of Section 20.2) combines channels, at cost $C_{in} \cdot C_{out}$. Figure 20.4.1 contrasts the two, and the illustration below tells the same story as two relaxed specialists splitting one overworked job.

The cost ratio of separable to standard is $\frac{K^2 C_{in} + C_{in} C_{out}}{K^2 C_{in} C_{out}} = \frac{1}{C_{out}} + \frac{1}{K^2}$. With a typical $K = 3$ and a few hundred output channels, the second term dominates and the reduction is about $1/9$, an order-of-magnitude saving for a small accuracy cost. In PyTorch, a depthwise convolution is simply a Conv2d with groups equal to the channel count.

# MobileNet's core unit splits a standard convolution into two cheap steps:
# a depthwise 3x3 (one filter per channel, groups=in_ch) does the spatial work,
# then a pointwise 1x1 mixes channels. Together they cost about 1/9 of the dense op.
import torch
import torch.nn as nn

class DepthwiseSeparable(nn.Module):
    """MobileNet's core unit: depthwise 3x3 then pointwise 1x1, each with BN+ReLU."""
    def __init__(self, in_ch, out_ch, stride=1):
        super().__init__()
        self.block = nn.Sequential(
            # depthwise: groups=in_ch means one KxK filter per channel
            nn.Conv2d(in_ch, in_ch, 3, stride=stride, padding=1,
                      groups=in_ch, bias=False),
            nn.BatchNorm2d(in_ch), nn.ReLU(inplace=True),
            # pointwise: 1x1 conv mixes channels and changes their count
            nn.Conv2d(in_ch, out_ch, 1, bias=False),
            nn.BatchNorm2d(out_ch), nn.ReLU(inplace=True),
        )

    def forward(self, x):
        return self.block(x)

dw = DepthwiseSeparable(64, 128)
std = nn.Conv2d(64, 128, 3, padding=1)
n_dw = sum(p.numel() for p in dw.parameters())
n_std = sum(p.numel() for p in std.parameters())
print(f"separable: {n_dw:,} params, standard: {n_std:,} params")

MobileNetV2 added a twist called the inverted residual. A normal ResNet bottleneck squeezes channels, works, then expands. The inverted residual does the opposite: it expands to a wide intermediate representation, applies the cheap depthwise convolution there, then projects back down, and the residual skip connects the two narrow ends. Doing the spatial work in the wide space preserves information, while keeping the skip-connected tensors thin saves memory, a clever rearrangement of the same parts. Figure 20.4.2 places the two side by side so you can read the inversion directly: where the channel width pinches, and which ends the skip connects.

2. ShuffleNet: Grouping, Then Mixing Advanced

ShuffleNet pushes the cost-cutting further with grouped convolutions, where the input channels are split into $g$ groups and each group is convolved independently, cutting the $1 \times 1$ pointwise cost by a factor of $g$. The danger is that if every layer groups the same way, information never crosses between groups and the network fragments into $g$ independent sub-networks. ShuffleNet's fix is a parameter-free channel shuffle: after a grouped convolution, permute the channels so that the next group draws from all previous groups. Information flows across the whole width at zero extra cost, the same "cheap operation that restores a property an optimization broke" pattern you saw in ResNet's skip.

# Channel shuffle is a parameter-free reshape-transpose-reshape that interleaves
# the channels of g groups. After a grouped convolution it lets the next grouped
# conv draw from every previous group, restoring cross-group information flow.
import torch

def channel_shuffle(x, groups: int):
    """Permute channels so the next grouped conv mixes across all groups."""
    b, c, h, w = x.shape
    x = x.view(b, groups, c // groups, h, w)  # split channels into groups
    x = x.transpose(1, 2).contiguous()        # swap group and per-group axes
    return x.view(b, c, h, w)                 # flatten back, now interleaved

x = torch.arange(8).view(1, 8, 1, 1).float()
print(x.flatten().tolist())
print(channel_shuffle(x, groups=4).flatten().tolist())

3. Squeeze-and-Excitation: Cheap Attention on Channels Advanced

A third efficiency idea, used inside MobileNetV3 and EfficientNet, is the squeeze-and-excitation (SE) block. It learns a per-channel importance weight: squeeze each feature map to a single number with global average pooling (averaging over the whole map is what gives each gate global context, so a channel is judged by how strongly it fires across the entire image rather than at one location), pass the resulting channel vector through a tiny two-layer network ending in a sigmoid, which squashes each output into a gate in $[0, 1]$ per channel, and multiply each channel by its gate. For almost no compute, the network learns to amplify informative channels and suppress noisy ones, a lightweight form of the attention you will study fully in Chapter 22. SE is one of the highest accuracy-per-FLOP additions known, which is why it appears in nearly every efficient design after 2018.

4. EfficientNet: Scale Everything Together Intermediate

The earlier designs cut the cost of a fixed network. EfficientNet asks the complementary question: given more compute, how should you grow a network? You can make it deeper (more layers), wider (more channels), or feed it higher-resolution images, and the field had historically scaled one of these at a time, by intuition. EfficientNet's compound scaling argues that the three should grow together in a fixed ratio, governed by a single coefficient $\phi$:

Here $\alpha$, $\beta$, and $\gamma$ are constants greater than one (the paper found roughly $\alpha = 1.2$, $\beta = 1.1$, $\gamma = 1.15$ by a small grid search), and $\phi$ is the single knob you turn to grow the network; setting $\phi = 0$ recovers the base network, and each step up multiplies depth, width, and resolution by those fixed per-step factors. The constraint keeps the total compute growing by roughly $2^\phi$ as $\phi$ increases, so each unit of $\phi$ doubles the budget and spreads it across all three dimensions in the empirically best proportion.

The squared exponents on width and resolution are not arbitrary; they fall straight out of how convolution counts cost. Doubling the depth stacks twice as many layers, so cost doubles, a factor of $2$. Doubling the width doubles both the input and the output channel count of every layer, and a convolution's cost scales with their product, so cost quadruples, a factor of $4$. Doubling the input resolution doubles both the height and the width of every feature map, so there are four times as many pixels to convolve, again a factor of $4$. Width and resolution each cost like an area (two dimensions growing at once) while depth costs like a length, which is exactly why they sit under squares in $\alpha \cdot \beta^2 \cdot \gamma^2$ and depth does not. The constraint is just the statement "whatever combination you pick, let the total compute roughly double per step of $\phi$".

With the scaling rule fixed, the remaining question is what to scale from. The base network is EfficientNet-B0, built from inverted-residual SE blocks and found by neural architecture search, an automated procedure that searches over candidate layer configurations to optimize accuracy under a cost budget rather than designing them by hand. Increasing $\phi$ from that base generates the B1 through B7 family, which traced an accuracy-per-FLOP frontier that dominated for years. The lesson reframes the depth-versus-width debate of Section 20.2: the answer is not depth or width but a principled mixture, plus resolution.

# Load the efficient backbones pretrained on ImageNet, picking by their
# accuracy-latency tradeoff. The depthwise grouping, inverted-residual and SE
# blocks, and channel-multiplier settings of this section are all inside timm.
import timm
# Pick a model by its accuracy-latency tradeoff, all pretrained on ImageNet:
mbv3 = timm.create_model("mobilenetv3_large_100", pretrained=True)  # ~5.5M params
effb0 = timm.create_model("efficientnet_b0", pretrained=True)       # ~5.3M params
# timm exposes FLOP and parameter stats for ranking candidates:
print(sum(p.numel() for p in effb0.parameters()))