04 · Gradient Gotchas 🪤

5 min read

A few autograd behaviors surprise almost everyone the first time. Learn them here and you'll write a correct training loop the first time in Module 05.

1. The 'Why'

Autograd is beautifully automatic, but a handful of its design decisions are counter-intuitive until someone explains them — and each one, misunderstood, produces a bug that's maddening to track down because nothing errors out. The most important is gradient accumulation: when you call .backward(), PyTorch adds the new gradients to whatever is already in .grad rather than replacing it. This is a deliberate, useful feature (it lets you sum gradients across several passes), but if you don't know about it, your gradients silently pile up across training steps and your model learns garbage.

The other gotchas — the difference between leaf and non-leaf tensors, and the graph being freed after one backward pass — round out the mental model you need before writing real training code. Getting these straight now means the training loop in Module 05 won't feel like a sequence of magic incantations; you'll know why optimizer.zero_grad() exists and why it must come at the right moment. We'll close by connecting everything you've learned about tracked tensors to the nn.Parameter objects that populate real models, which is the bridge into Module 03.

2. Core Concepts

Gradients accumulate. Each .backward() call adds into .grad. To get the gradient for this step alone, you must zero it first — x.grad.zero_() by hand, or optimizer.zero_grad() in a real loop.

Leaf vs. non-leaf, revisited. Only leaf tensors (requires_grad=True, created by you, not by an op) retain .grad by default. Intermediates don't, unless you call .retain_grad().

The graph is freed after backward. By default PyTorch discards the graph once you've backpropagated through it, to save memory. A second .backward() on the same graph errors unless you passed retain_graph=True (needed only in special cases like multiple losses sharing a graph).

From tensors to parameters (the bridge to Module 03). Everything here operates on tensors you made by hand. In real models you'll rarely do that — instead, layers hold nn.Parameter objects, which are literally just tensors with requires_grad=True that a module registers and tracks for you. The autograd rules don't change one bit; they're simply managed automatically.

3. Code in Action

Watching gradients accumulate (the classic trap)

import torch

x = torch.tensor(2.0, requires_grad=True)

for step in range(3):
    y = x ** 2               # dy/dx = 2x = 4
    y.backward()             # ADDS 4 into x.grad each iteration
    print(x.grad)            # tensor(4.) then tensor(8.) then tensor(12.)  ⚠️

The fix: zero the gradient each step

import torch

x = torch.tensor(2.0, requires_grad=True)

for step in range(3):
    if x.grad is not None:
        x.grad.zero_()       # reset to 0 BEFORE this step's backward
    y = x ** 2
    y.backward()
    print(x.grad)            # tensor(4.) every time  ✓

`retain_grad` and `retain_graph`

import torch

x = torch.tensor(3.0, requires_grad=True)
y = x * 2                    # y is a NON-leaf intermediate
y.retain_grad()             # ask PyTorch to keep y's gradient too
z = y + 1

z.backward(retain_graph=True)  # keep the graph so we can backprop again if needed
print(x.grad)               # tensor(2.) -> dz/dx
print(y.grad)               # tensor(1.) -> dz/dy (only available thanks to retain_grad)

Preview: the same rules, wrapped in `nn.Parameter`

import torch
import torch.nn as nn

# An nn.Parameter is just a tensor with requires_grad=True that a module tracks.
p = nn.Parameter(torch.randn(3))   # requires_grad is True automatically
print(p.requires_grad)             # True

linear = nn.Linear(3, 1)           # a real layer full of Parameters
for name, param in linear.named_parameters():
    print(name, param.requires_grad)   # weight True / bias True -> autograd, managed for you

4. Common Pitfalls

Pitfall 1 — Forgetting to zero gradients. Because .backward() accumulates, skipping zero_grad() makes every step's gradient the sum of all previous steps. Training destabilizes and you'll have no error message to guide you. In Module 05 this becomes the reflexive first line of the loop.

Pitfall 2 — Zeroing at the wrong time. Zero before the backward pass of the current step, not after the optimizer step in a way that wipes gradients you still need. The canonical order is: zero_grad() → forward → loss.backward() → optimizer.step().

Pitfall 3 — Reaching for retain_graph=True to silence an error. Beginners often add it to fix "Trying to backward through the graph a second time." Usually the real bug is calling .backward() twice by accident, or reusing a stale loss. Only use retain_graph=True when you genuinely need multiple backward passes over one graph.

5. Further Reading & Watch List

⬅️ Prev: 03 · Turning Autograd Off · 🏁 Module complete! Back to the module index or on to Module 03 — Neural Networks.