04 · Serving & Next Steps 🎓

The last mile: wrapping your model in an API someone can call — and a map for where to take your PyTorch journey from here.

---

1. The 'Why'

You've exported your model, compiled it, and optimized it for efficient inference. There's one final gap to close: making it callable by the outside world. In most products, a model doesn't run in the same process as the user — it lives behind a service, an HTTP API that accepts input (an image, some text, a feature vector), runs inference, and returns a prediction. Understanding this last mile matters because it's where all your work becomes a product, and because the shape of the serving layer influences everything upstream: how you export, how you batch, what latency budget you have. Even a minimal example demystifies the whole picture — a trained model is, at serving time, just a function you expose over the network.

This sub-module also serves as the capstone of the entire pytorch-101 course. You began not knowing what a tensor was; you can now build, train, evaluate, save, optimize, and deploy real deep learning models across vision and language, with and without a framework. That's a genuinely strong foundation — the same foundation the largest models are built on. The final thing a good mentor does is point you toward the horizon: the vibrant ecosystem and advanced topics that build directly on what you now know. So the second half of this sub-module is a curated "where to go next," so you leave with momentum rather than a full stop.

---

2. Core Concepts

A model server is a function behind HTTP. At its simplest: load the model once at startup (in eval()), accept a request, convert its payload to a tensor, run inference under no_grad(), and return the result as JSON. Everything you learned about inference mode applies unchanged.

Load once, not per request. Loading the model on every request is slow and wasteful. Load it a single time when the service starts and reuse it — a common beginner serving mistake.

Batching for throughput. Under load, grouping concurrent requests into a batch uses hardware far more efficiently (Module 04's batching, applied at serve time). Frameworks like TorchServe do this automatically.

Purpose-built serving tools. Beyond a hand-rolled API, TorchServe, ONNX Runtime, Triton, and cloud endpoints handle batching, scaling, versioning, and monitoring for production workloads.

Where to go next. The ecosystem — Hugging Face, distributed training, generative models, RL — all builds on the fundamentals from this course.

---

3. Code in Action

A minimal model-serving API (FastAPI)

# app.py — a tiny inference service (run with: uvicorn app:app)
import torch
import torch.nn as nn
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

# Load the model ONCE at startup, in eval mode
model = nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 3))
model.load_state_dict(torch.load("best_model.pt", map_location="cpu"))
model.eval()

class Request(BaseModel):
    features: list[float]          # e.g. [1.0, 2.0, 3.0, 4.0]

@app.post("/predict")
def predict(req: Request):
    x = torch.tensor(req.features).unsqueeze(0)   # add batch dim (Module 01)
    with torch.no_grad():                         # inference mode (Module 06)
        logits = model(x)
        pred = int(logits.argmax(1).item())
    return {"prediction": pred}                    # returned as JSON

Calling the service

curl -X POST http://localhost:8000/predict \
     -H "Content-Type: application/json" \
     -d '{"features": [1.0, 2.0, 3.0, 4.0]}'
# -> {"prediction": 2}

A production-readiness checklist

[ ] Model exported/optimized appropriately for the target hardware (subs 01–03)
[ ] Loaded once at startup, in eval() mode, under no_grad()
[ ] Input validation + preprocessing matches TRAINING (same transforms/normalization)
[ ] Batching enabled if throughput matters
[ ] Accuracy re-verified after any quantization/precision change
[ ] Versioning + monitoring for the deployed model

---

4. Common Pitfalls

Pitfall 1 — Preprocessing mismatch between training and serving. The #1 real-world deployment bug: the server normalizes/resizes inputs differently than training did, so a model that scored well silently makes poor predictions in production. Reuse the exact transforms and normalization constants from training (Modules 04, 07).

Pitfall 2 — Reloading the model on every request. Calling torch.load inside the request handler adds huge latency and memory churn. Load once at startup and hold the model in memory for the life of the service.

Pitfall 3 — Treating the course as finished learning. Deep learning moves fast; the fundamentals you have are the entry ticket, not the destination. The biggest mistake now would be to stop. Pick a next topic below and build something.

🚀 Where to go next

• Hugging Face — transformers and datasets put thousands of pretrained models (including the Transformers from Module 08) one line away. The fastest route to state-of-the-art NLP and vision.
• Distributed & large-scale training — DistributedDataParallel, FSDP, and Lightning's multi-GPU support (Module 09) for training bigger models on more hardware.
• Generative models — diffusion models (images), and building/fine-tuning LLMs, both direct extensions of Modules 07–08.
• Reinforcement learning — a different training paradigm where models learn from reward rather than labels.
• Keep building — the best next step is a project of your own. Re-implement a paper, enter a Kaggle competition, or ship the FastAPI service above.

---

5. Further Reading & Watch List

• 📖 Deep Learning with PyTorch, Ch. 15 — serving models
• 🌐 Hugging Face course · Learn PyTorch — where to go next
• 🛠️ TorchServe · FastAPI · Official PyTorch deployment tutorials
• 🎓 fast.ai — Part 2: from foundations to state-of-the-art · ▶️ Karpathy — Neural Networks: Zero to Hero

---

⬅️ Prev: 03 · Inference Optimization · 🏁 You've completed pytorch-101! 🎉 Head back to the root README to review the full roadmap — or better yet, go build something.