03 · Turning Autograd Off 🛑

4 min read

Tracking gradients is essential during training — and pure overhead everywhere else. Knowing how to switch it off is a mark of a competent PyTorch user.

1. The 'Why'

Autograd's graph-building is the machinery of learning, but it isn't free. Every tracked operation stores extra information so the backward pass can run later, which costs both memory and time. During training that cost is worth it. But there are large stretches of a deep learning workflow where you have no intention of calling .backward() at all — running a trained model to make predictions (inference), measuring accuracy on a validation set (evaluation), or manually tweaking weights. In all of these, building a graph is wasted effort, and on a big model it can be the difference between fitting in memory and crashing with an out-of-memory error.

There's a second, subtler reason beyond performance: correctness. If you update a model's weights inside a tracked context, those updates themselves become part of the graph, and PyTorch will try to backpropagate through your optimization step — which is almost never what you want and produces confusing bugs. Turning autograd off draws a clean line between "computation I want to learn from" and "computation I just want the answer to." This sub-module covers the two tools for doing that: the torch.no_grad() context manager for whole blocks of code, and .detach() for surgically removing a single tensor from the graph.

2. Core Concepts

torch.no_grad(). A context manager. Any operation inside a with torch.no_grad(): block is not recorded — the results have requires_grad=False and carry no grad_fn. Use it to wrap inference, evaluation, and manual weight updates.

.detach(). Returns a new tensor that shares the same data but is cut off from the graph. Use it when you need to pull a value out of a tracked computation — for logging, converting to NumPy, or feeding into code that shouldn't backpropagate.

no_grad() vs. .detach(). no_grad() disables tracking for a region of code; .detach() disconnects a single tensor. Reach for the context manager around blocks, and .detach() for one-off values.

The model.eval() distinction. Note that torch.no_grad() is about gradients, which is separate from model.eval() (Module 05), which changes layer behavior like Dropout and BatchNorm. Real evaluation code typically uses both.

3. Code in Action

`torch.no_grad()` for inference

import torch

w = torch.randn(3, requires_grad=True)
x = torch.tensor([1.0, 2.0, 3.0])

with torch.no_grad():            # nothing inside here is recorded
    pred = (w * x).sum()         # a normal computation...
    print(pred.requires_grad)    # False -> no graph was built

# Outside the block, tracking resumes as normal
pred2 = (w * x).sum()
print(pred2.requires_grad)       # True

`.detach()` to pull a value out of the graph

import torch

w = torch.randn(3, requires_grad=True)
loss = (w ** 2).sum()            # tracked scalar

# Grab a plain copy for logging without dragging the graph along
loss_value = loss.detach()       # same number, but requires_grad=False
print(loss_value.requires_grad)  # False

# Detaching is also required before going to NumPy on a tracked tensor
as_numpy = loss.detach().cpu().numpy()   # detach -> cpu -> numpy

The manual weight-update pattern

import torch

w = torch.tensor([1.0], requires_grad=True)
loss = (w - 5) ** 2              # a toy loss with a minimum at w = 5
loss.backward()                  # w.grad now holds dLoss/dw

with torch.no_grad():            # the update itself must NOT be tracked
    w -= 0.1 * w.grad            # gradient-descent step on the raw values
print(w)                         # moved toward 5

4. Common Pitfalls

Pitfall 1 — Updating weights without no_grad(). Modifying a tracked tensor in place outside a no_grad() block makes the update part of the graph and raises errors (or silently corrupts training). Always wrap manual parameter updates in torch.no_grad(). (In practice an optimizer handles this for you — see Module 05.)

Pitfall 2 — Confusing no_grad() with eval(). They solve different problems. torch.no_grad() stops gradient tracking; model.eval() switches Dropout/BatchNorm into inference mode. Using only one when you need both gives wrong or slow results. Evaluation loops should use both.

Pitfall 3 — Forgetting to .detach() before .numpy(). Calling .numpy() on a tensor that still requires grad raises "Can't call numpy() on Tensor that requires grad." The fix is tensor.detach().cpu().numpy().

5. Further Reading & Watch List

⬅️ Prev: 02 · Backward Pass & Gradients · ➡️ Next: 04 · Gradient Gotchas