03 · Inference Optimization 🪶

5 min read

A model can be made smaller and faster without retraining it. When every millisecond and megabyte counts, these techniques are how you fit a model onto real hardware.

1. The 'Why'

Training optimizes a model for accuracy; deployment forces you to also care about cost — latency, memory, and compute. A model that takes 200ms per prediction might be unusable in a real-time app; one that needs 2GB of RAM won't fit on a phone or a cheap server; a high-traffic API multiplies every inefficiency by millions of requests. The good news is that a trained model usually has slack: it was trained in 32-bit precision with plenty of numerical headroom, and much of that precision is unnecessary at inference time. Inference optimization exploits that slack to make the model dramatically leaner and faster while giving up little or no accuracy — often without retraining at all.

The two most accessible levers are precision reduction and quantization. Neural networks are surprisingly robust to using fewer bits per number: switching from 32-bit floats to 16-bit half precision roughly halves memory and speeds up math on modern GPUs, typically with negligible accuracy loss. Quantization goes further, converting weights (and sometimes activations) to 8-bit integers, shrinking the model ~4× and enabling fast integer arithmetic — especially valuable on CPUs and edge devices. Beyond the model itself, serving efficiently means batching requests, keeping the model in eval()/no_grad(), and choosing the right runtime. This sub-module gives you the conceptual map and the first tools; the field goes deep, but these fundamentals cover most real needs.

2. Core Concepts

Precision reduction (FP16/BF16). Store and compute in 16-bit floats instead of 32-bit. Roughly halves memory and accelerates matrix math on capable GPUs, usually with minimal accuracy impact. Easy to apply for inference via .half() or autocast.

Quantization (INT8). Map float weights/activations to 8-bit integers. Shrinks the model ~4× and enables fast integer kernels. Flavors: dynamic (quantize weights ahead of time, activations on the fly — easiest, great for LSTMs/Transformers on CPU), static (calibrate with sample data for more speed), and quantization-aware training (simulate quantization during training for best accuracy).

The accuracy/efficiency trade-off. More aggressive compression (INT8 static) yields more speed but risks more accuracy loss than FP16 or dynamic quantization. Always measure accuracy after optimizing.

Serving fundamentals. Batch incoming requests to use hardware efficiently, keep the model in eval() + no_grad() (Module 06), warm up compiled/quantized models, and pick a runtime (TorchServe, ONNX Runtime, etc.) suited to your hardware.

3. Code in Action

Half precision for GPU inference

import torch
import torch.nn as nn

model = nn.Linear(1024, 10).eval().cuda()   # a model on the GPU, in eval mode

# Option A: cast the whole model + input to FP16
half_model = model.half()                    # weights -> float16
x = torch.randn(64, 1024, device="cuda").half()
with torch.no_grad():
    out = half_model(x)                      # ~2x less memory, faster matmul

# Option B: autocast — let PyTorch choose precision per op (often safer)
with torch.no_grad(), torch.autocast(device_type="cuda", dtype=torch.float16):
    out = model(torch.randn(64, 1024, device="cuda"))

Dynamic quantization (great for CPU, LSTMs/Transformers/Linear-heavy models)

import torch
import torch.nn as nn

model = nn.Sequential(nn.Linear(1024, 1024), nn.ReLU(), nn.Linear(1024, 10)).eval()

quantized = torch.quantization.quantize_dynamic(
    model,
    {nn.Linear},              # quantize Linear layers to INT8
    dtype=torch.qint8,
)
# ~4x smaller, faster integer math on CPU — no retraining, no calibration data.
x = torch.randn(8, 1024)
with torch.no_grad():
    print(quantized(x).shape)   # torch.Size([8, 10])

Always measure the accuracy cost

import torch

@torch.no_grad()
def accuracy(model, loader, device="cpu"):
    model.eval(); correct = total = 0
    for X, y in loader:
        X, y = X.to(device), y.to(device)
        correct += (model(X).argmax(1) == y).sum().item()
        total += y.size(0)
    return correct / total

# base = accuracy(model, test_loader)
# quant = accuracy(quantized, test_loader)
# print(f"fp32 {base:.3f} vs int8 {quant:.3f}")   # confirm the drop is acceptable

4. Common Pitfalls

Pitfall 1 — Optimizing without measuring accuracy. Quantization and precision cuts can degrade accuracy, sometimes sharply for a sensitive model. Never ship an optimized model on faith — evaluate it on your test set (Module 05) and compare against the FP32 baseline before deploying.

Pitfall 2 — Expecting FP16 speedups on a CPU. Half precision accelerates math on GPUs with FP16 support; on many CPUs it offers little or is even slower. For CPU deployment, reach for INT8 quantization instead — match the technique to the target hardware.

Pitfall 3 — Forgetting eval() + no_grad() when benchmarking or serving. Leaving Dropout active or building autograd graphs inflates latency and memory and corrupts predictions — and makes your optimization measurements meaningless. Inference always runs in eval() under no_grad() (Module 06's recipe).

5. Further Reading & Watch List

⬅️ Prev: 02 · torch.compile · ➡️ Next: 04 · Serving & Next Steps