04 · Evaluation & Metrics 📊

5 min read

Loss is what the model optimizes; metrics are what you actually care about. And the gap between training and validation curves tells you whether your model is really learning.

1. The 'Why'

The loss function is a mathematical convenience — it's smooth and differentiable so the optimizer can work with it — but it's rarely the number a human cares about. Nobody ships a product because the cross-entropy hit 0.31; they ship it because it classifies images correctly 94% of the time. That human-meaningful number is a metric, and computing metrics like accuracy is how you translate training progress into a claim you can actually stand behind. A model can have a respectable-looking loss and still be useless, so tracking the right metric on held-out data is what separates "the loss went down" from "the model works."

Just as important is watching how metrics evolve on training versus validation data. This is your window into the two failure modes that haunt every deep learning project. Overfitting shows up as training loss that keeps falling while validation loss stalls or rises — the model is memorizing the training set instead of learning general patterns. Underfitting shows up as both staying stubbornly high — the model isn't capable or trained enough. Learning to read these curves turns training from guesswork into diagnosis. And it sets up Module 06 perfectly: once you can tell which epoch had the best validation performance, you'll want to save that exact model — the checkpoint problem.

2. Core Concepts

Loss vs. metric. The loss is optimized by gradient descent (must be differentiable). A metric (accuracy, F1, etc.) is for human interpretation and can be non-differentiable. You compute both, but only the loss goes into .backward().

Accuracy for classification. The fraction of predictions that match the true label. Get the predicted class with logits.argmax(dim=1), compare to targets, and average the matches.

Evaluate under eval() + no_grad(). All metric computation happens with the model in eval() mode and inside torch.no_grad() — you're measuring, not learning.

Overfitting vs. underfitting (reading the curves). Overfitting: training keeps improving, validation plateaus or worsens (a widening gap). Underfitting: both are poor. The goal is the sweet spot where validation is as good as it gets.

Tracking history. Store per-epoch train and validation numbers in lists so you can print or plot the learning curves and pick the best epoch.

3. Code in Action

Computing accuracy

import torch

def accuracy(logits, targets):
    preds = logits.argmax(dim=1)             # predicted class = highest logit
    correct = (preds == targets).sum().item()# count matches
    return correct / targets.size(0)         # fraction correct

logits = torch.tensor([[2.0, 0.1, 0.3],      # sample 0 -> predicts class 0
                       [0.2, 0.1, 1.9]])     # sample 1 -> predicts class 2
targets = torch.tensor([0, 1])               # true: class 0, class 1
print(accuracy(logits, targets))             # 0.5 -> one of two correct

A full evaluation pass

import torch

@torch.no_grad()                             # decorator = wrap whole fn in no_grad
def evaluate(model, loader, criterion, device):
    model.eval()                             # inference mode
    total_loss, total_correct, total = 0.0, 0, 0
    for X, y in loader:
        X, y = X.to(device), y.to(device)
        logits = model(X)
        total_loss += criterion(logits, y).item() * y.size(0)   # sum, weighted by batch
        total_correct += (logits.argmax(1) == y).sum().item()
        total += y.size(0)
    return total_loss / total, total_correct / total            # (avg loss, accuracy)

Tracking history to spot overfitting

history = {"train_loss": [], "val_loss": [], "val_acc": []}

for epoch in range(20):
    train_loss = train_one_epoch(model, train_loader, criterion, optimizer, device)
    val_loss, val_acc = evaluate(model, val_loader, criterion, device)

    history["train_loss"].append(train_loss)
    history["val_loss"].append(val_loss)
    history["val_acc"].append(val_acc)
    print(f"epoch {epoch}: train {train_loss:.3f} | val {val_loss:.3f} | acc {val_acc:.3f}")

# Overfitting signal: train_loss keeps dropping while val_loss starts climbing.
best_epoch = min(range(len(history["val_loss"])), key=lambda i: history["val_loss"][i])
print(f"Best epoch by val loss: {best_epoch}")   # <-- this motivates saving a checkpoint (Module 06)

4. Common Pitfalls

Pitfall 1 — Judging the model by training accuracy. High training accuracy can just mean memorization. Always report the metric on a held-out validation/test set; that's the only number that reflects generalization.

Pitfall 2 — Computing metrics with gradients on. Forgetting eval() and no_grad() during evaluation wastes memory, slows things down, and (without eval()) gives noisy numbers because Dropout is still active. Wrap every evaluation in both.

Pitfall 3 — Ignoring the train/validation gap. Watching only one curve hides the diagnosis. A model with 0.99 train accuracy and 0.70 validation accuracy is overfitting badly — the gap is the story. Track both every epoch so you can catch it early (and later, stop or checkpoint at the best point).

5. Further Reading & Watch List

⬅️ Prev: 03 · The Training Loop · 🏁 Module complete! Back to the module index or on to Module 06 — Saving & Loading.