Background
Back to Articles/PyTorch 101: Zero to Hero
Sections
IntroductionModule 01 — Tensors 🧊01 · Creating Tensors 🧊02 · Indexing & Reshaping 🔪03 · Tensor Math ➗04 · Device Placement 🖥️⚡Module 02 — Autograd ⚙️01 · The Computational Graph 🕸️02 · Backward Pass & Gradients ⬅️03 · Turning Autograd Off 🛑04 · Gradient Gotchas 🪤Module 03 — Neural Networks01 · The `nn.Module` Basics 🧱02 · Common Layers 🧩03 · Building a Network 🏗️04 · Inspecting Models 🔍Module 04 — Data Handling 🗂️01 · Dataset Basics 📇02 · The DataLoader 🚚03 · Transforms 🎨04 · Splits & Built-in Datasets ✂️Module 05 — The Training Loop 🔁01 · Loss Functions 🎯02 · Optimizers ⚙️03 · The Training Loop 🔁04 · Evaluation & Metrics 📊Module 06 — Saving & Loading 💾01 · `state_dict` Basics 💾02 · Checkpoints & Resuming ⏸️03 · Loading for Inference 🚀04 · Best Model & Early Stopping 🏅Module 07 — Computer Vision 👁️01 · Convolutions 🔲02 · CNN Architecture 🏛️03 · Transfer Learning 🔁04 · Image Classification Project 🧪Module 08 — NLP & Transformers 💬01 · Text Data & Tokenization 🔤02 · Embeddings 🧭03 · Recurrent Layers & LSTMs 🔄04 · Intro to Transformers ⚡Module 09 — Ecosystem: PyTorch Lightning ⚡01 · Why Lightning? 🤔02 · The LightningModule 🧩03 · The Trainer 🎛️04 · DataModules & Callbacks 🧰Module 10 — Deployment & Optimization 🚀01 · Exporting Models 📦02 · `torch.compile` ⚡03 · Inference Optimization 🪶04 · Serving & Next Steps 🎓

Module 02 — Autograd ⚙️

2 min read

The engine that makes neural networks learn. Once you see how it works, "training a model" stops feeling like magic.

In Module 01 you learned that a tensor is a supercharged NumPy array. Here's the superpower that NumPy can't touch: a PyTorch tensor can remember every operation performed on it, then automatically compute the gradients — the derivatives — of some final output with respect to every input. That process is called automatic differentiation, and PyTorch's implementation is autograd.

Why does this matter? Training a neural network means repeatedly nudging millions of weights in the direction that reduces error. To know which direction, you need the gradient of the loss with respect to each weight. Computing those derivatives by hand would be hopeless. Autograd does it for you: you write the forward computation as ordinary Python, call .backward(), and every gradient appears. Master this module and the training loop in Phase 2 will feel almost inevitable.

This module is split into three sub-modules — work through them in order.

📚 Sub-Modules

# Sub-Module What you'll learn
01 The Computational Graph How PyTorch records operations into a dynamic graph, and the role of requires_grad
02 Backward Pass & Gradients The forward vs. backward pass, calling .backward(), and reading gradients from .grad
03 Turning Autograd Off torch.no_grad(), .detach(), and why inference and evaluation need them
04 Gradient Gotchas Why .grad accumulates (and must be zeroed), leaf vs. non-leaf tensors, and retain_graph

🎯 By the end of this module, you'll be able to...

  • Explain what the computational graph is and how requires_grad puts a tensor on it.
  • Run a forward pass, call .backward(), and interpret the gradients stored in .grad.
  • Correctly disable gradient tracking for inference to save memory and avoid subtle bugs.
  • Understand why gradients accumulate and how zeroing them prepares you for the training loop.

✅ Prerequisites

Comfort with tensors from Module 01 — creating them, doing math, and moving them across devices.

⬅️ Prev module: 01 · Tensors · ➡️ Next module: 03 · Neural Networks