01 · The Computational Graph 🕸️

4 min read

Autograd's secret is that every operation you perform is quietly being recorded. That recording is the computational graph.

1. The 'Why'

Training a neural network boils down to one question asked millions of times: "If I nudge this weight a tiny bit, does the error go up or down, and by how much?" The answer is a derivative — a gradient. A modern network has millions of weights, and the error depends on all of them through a long chain of operations. Working out those derivatives by hand, using the chain rule from calculus, would be impossibly tedious and error-prone. We need the computer to do it for us, automatically, every single training step.

That's exactly what autograd provides, and the computational graph is how it works. As you perform operations on tensors, PyTorch silently builds a graph in the background: each tensor is a node, and each operation records how its output was produced from its inputs. When you later ask for gradients, PyTorch walks backward through this graph, applying the chain rule mechanically at each step. The beautiful part is that the graph is built dynamically — it's just ordinary Python running forward — so you can use loops, conditionals, and any control flow you like, and the graph reflects exactly what ran. Understanding this graph is the key that unlocks everything else in this module.

2. Core Concepts

requires_grad. A tensor is only tracked if it opts in. Set requires_grad=True (or create it via an operation on a tracked tensor) and PyTorch starts recording. Weights in a real model have this on; raw input data usually doesn't.

The graph is dynamic ("define-by-run"). Unlike older frameworks that compiled a static graph up front, PyTorch builds the graph as your code executes. Each forward pass creates a fresh graph. This is why debugging PyTorch feels like debugging normal Python.

grad_fn. Every tensor produced by a tracked operation carries a grad_fn — a reference to the function that created it (e.g. AddBackward, MulBackward). This is the backward "recipe" for that node. Leaf tensors you created yourself have grad_fn=None.

Leaf vs. non-leaf. Tensors you create with requires_grad=True are leaves (the things you ultimately want gradients for). Tensors produced by operations are non-leaf intermediates. This distinction matters in sub-module 04.

3. Code in Action

Opting a tensor into the graph

import torch

x = torch.tensor([2.0, 3.0], requires_grad=True)  # opt in to autograd tracking
print(x.requires_grad)   # True  -> PyTorch will record ops on x
print(x.grad_fn)         # None  -> x is a leaf; nothing created it

w = torch.tensor([4.0])  # requires_grad defaults to False
print(w.requires_grad)   # False -> ordinary tensor, not tracked

Watching the graph get built

import torch

x = torch.tensor(3.0, requires_grad=True)  # a tracked leaf
y = x * 2                                   # an operation -> creates a graph node
z = y + 1                                   # another operation -> another node

print(y.grad_fn)   # <MulBackward0 ...> -> records that y came from a multiply
print(z.grad_fn)   # <AddBackward0 ...> -> records that z came from an add
# The chain z <- y <- x is the computational graph PyTorch will walk backward.

Turning tracking on after creation

import torch

a = torch.ones(2, 2)              # created untracked
a.requires_grad_(True)            # in-place: flip tracking ON (note trailing _)
b = (a * 3).sum()                 # b is now part of a graph rooted at a
print(b.grad_fn)                  # <SumBackward0 ...>

4. Common Pitfalls

Pitfall 1 — Expecting .grad before a backward pass. Building the graph does not compute gradients. x.grad stays None until you call .backward() (sub-module 02). The graph is just the recording; the gradients come later.

Pitfall 2 — Assuming inputs are tracked. Raw data tensors default to requires_grad=False. If you build something expecting gradients to flow to an input and forget to enable tracking, .backward() silently produces no gradient for it. Only leaves with requires_grad=True accumulate .grad.

Pitfall 3 — Confusing requires_grad_() with requires_grad. The trailing underscore version is an in-place setter (a.requires_grad_(True)); the bare attribute is a read-only-looking flag you set at creation. Mixing them up leads to tensors you think are tracked but aren't.

5. Further Reading & Watch List

➡️ Next: 02 · Backward Pass & Gradients