03 · Tensor Math ➗

4 min read

Neural networks are, under the hood, a long chain of matrix multiplications and element-wise operations. This is where the "math" in "the math" lives.

1. The 'Why'

Strip away the jargon and a neural network is mostly two operations repeated over and over: matrix multiplication (combining inputs with learned weights) and element-wise functions (activations like ReLU applied to each number independently). If you're fuzzy on the difference between these, your models will produce nonsense and you won't know why — because both operations often run without error while giving completely different results. That silent failure mode is exactly why tensor math deserves careful attention.

Beyond the two headline operations, you'll constantly reduce tensors — summing a loss across a batch, averaging accuracy, finding the maximum-scoring class. And you'll rely on broadcasting, PyTorch's rule for automatically stretching mismatched shapes so you can, say, add a bias vector to every row of a matrix without writing a loop. Together these tools are the arithmetic backbone of everything you'll build, so it's worth building real intuition for them now rather than debugging them later.

2. Core Concepts

Element-wise operations (+, -, *, /) act position-by-position. A * B multiplies each element of A by the element in the same spot of B. Shapes must match or be broadcastable.

Matrix multiplication uses @ or torch.matmul. This is the linear-algebra product where inner dimensions must agree: a (2×3) can multiply a (3×4) to give a (2×4). This is not the same as *.

Reductions collapse dimensions: .sum(), .mean(), .max(), .argmax(). Add a dim= argument to reduce along a specific axis — dim=0 collapses rows (operating down columns), dim=1 collapses columns (operating across rows).

Broadcasting lets operations combine tensors of different shapes by virtually expanding size-1 (or missing) dimensions. Adding a shape-(3,) bias to a shape-(2, 3) matrix adds it to both rows.

3. Code in Action

Element-wise vs. matrix multiplication

import torch

A = torch.tensor([[1.0, 2.0],
                  [3.0, 4.0]])       # 2x2 matrix
B = torch.tensor([[5.0, 6.0],
                  [7.0, 8.0]])       # 2x2 matrix

print(A * B)             # ELEMENT-WISE product: multiplies matching positions
print(A @ B)             # MATRIX product: rows-times-columns (linear algebra)
print(torch.matmul(A, B))# identical to A @ B

# Inner dimensions must match for matmul:
X = torch.rand(2, 3)     # shape (2, 3)
Y = torch.rand(3, 4)     # shape (3, 4)
print((X @ Y).shape)     # torch.Size([2, 4]) -> inner 3s cancel

Reductions

import torch

A = torch.tensor([[1.0, 2.0],
                  [3.0, 4.0]])

print(A.sum())           # tensor(10.) -> sum of ALL elements
print(A.mean())          # tensor(2.5) -> mean of all elements
print(A.sum(dim=0))      # tensor([4., 6.]) -> sum down each COLUMN
print(A.sum(dim=1))      # tensor([3., 7.]) -> sum across each ROW
print(A.argmax())        # tensor(3) -> flat index of the largest value (the 4.0)

Broadcasting and transpose

import torch

matrix = torch.ones(2, 3)          # shape (2, 3), all ones
bias   = torch.tensor([1., 2., 3.])# shape (3,)

# bias is broadcast across BOTH rows of matrix — no loop needed
print(matrix + bias)     # each row becomes [2., 3., 4.]

# Transpose swaps two dimensions (rows <-> columns)
print(matrix.T.shape)    # torch.Size([3, 2])

4. Common Pitfalls

Pitfall 1 — Confusing * with @. A * B is element-wise; A @ B is matrix multiplication. Both may run silently, so a wrong operator gives a plausible-but-wrong result. When your loss looks bizarre, check this first.

Pitfall 2 — Matrix-multiply dimension mismatch. matmul needs inner dimensions to agree: (2×3) @ (3×4) works, (2×3) @ (4×3) doesn't. Print .shape on both operands, and remember you can .T one of them to line them up.

Pitfall 3 — The wrong reduction dim. dim=0 and dim=1 reduce different axes, and mixing them up quietly corrupts metrics like per-sample accuracy. Say it in words first: "I want one number per row" → reduce across columns → dim=1.

5. Further Reading & Watch List

⬅️ Prev: 02 · Indexing & Reshaping · ➡️ Next: 04 · Device Placement