02 · Backward Pass & Gradients ⬅️

4 min read

You've built the graph. Now watch PyTorch walk it backward and hand you every gradient for free.

1. The 'Why'

The whole point of recording a computational graph (sub-module 01) is to answer one question efficiently: how does the final output change as each input changes? In deep learning that output is the loss — a single number measuring how wrong the model is — and the inputs are the model's weights. The gradient of the loss with respect to each weight tells us which way to nudge that weight to reduce the error. Gather all those gradients and you have the exact recipe for improving the model. This backward computation is the beating heart of every training loop you'll ever write.

Doing it by hand means applying the chain rule across the entire graph, multiplying local derivatives together from the output all the way back to each input. Autograd automates this completely. You run your computation forward as normal Python (the forward pass), call .backward() on the final scalar, and PyTorch traverses the graph in reverse (the backward pass), depositing the gradient of that scalar with respect to every tracked leaf into its .grad attribute. One line of code replaces pages of calculus. Once this clicks, the training loop in Module 05 will feel like a natural consequence rather than a mystery.

2. Core Concepts

Forward pass. Running your operations to produce an output. This is where the graph gets built.

Backward pass. Calling .backward() on the output. PyTorch walks the graph from that output back to the leaves, applying the chain rule and accumulating results into each leaf's .grad.

The scalar rule. .backward() with no arguments only works on a scalar (a single number) — typically your loss. Gradients are "of one number with respect to many," so PyTorch needs a single starting point. For non-scalar outputs you must pass a gradient= argument (the vector-Jacobian product), which is rare in day-to-day training.

.grad. After .backward(), each leaf tensor with requires_grad=True has its gradient stored in .grad, a tensor of the same shape.

3. Code in Action

A minimal backward pass

import torch

x = torch.tensor(3.0, requires_grad=True)   # tracked leaf
y = x ** 2                                   # forward pass: y = x^2

y.backward()             # backward pass: compute dy/dx and store in x.grad
print(x.grad)            # tensor(6.) -> dy/dx = 2x = 2*3 = 6  ✓

Gradients with multiple inputs

import torch

a = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(4.0, requires_grad=True)

L = a * b + b ** 2       # forward: L = a*b + b^2  (think "loss")
L.backward()             # walk the graph backward from L

print(a.grad)            # tensor(4.)  -> dL/da = b        = 4
print(b.grad)            # tensor(10.) -> dL/db = a + 2b   = 2 + 8 = 10

The scalar rule in practice

import torch

w = torch.randn(3, requires_grad=True)   # a small "weight" vector
x = torch.tensor([1.0, 2.0, 3.0])        # input data (untracked)

pred = (w * x).sum()     # reduce to a SCALAR before calling backward
pred.backward()          # works because pred is a single number
print(w.grad)            # tensor([1., 2., 3.]) -> d(pred)/dw = x  ✓

# out = w * x            # a vector — out.backward() would raise:
#                        # "grad can be implicitly created only for scalar outputs"

4. Common Pitfalls

Pitfall 1 — Calling .backward() on a non-scalar. If your output is a vector or matrix, .backward() errors out. Reduce to a single number first (usually .sum() or .mean() — which is exactly what a loss function does).

Pitfall 2 — Reading .grad on a non-leaf tensor. Gradients are only retained for leaf tensors by default. Checking .grad on an intermediate value returns None (and warns). If you truly need an intermediate's gradient, call .retain_grad() on it before the backward pass.

Pitfall 3 — Expecting the graph to survive a second .backward(). By default the graph is freed after one backward pass to save memory, so a second .backward() on the same graph raises an error. This — and the fact that gradients accumulate rather than overwrite — is covered in sub-module 04.

5. Further Reading & Watch List

⬅️ Prev: 01 · The Computational Graph · ➡️ Next: 03 · Turning Autograd Off