03 · Building a Network 🏗️

Individual layers are Lego bricks. Now let's snap them together into a real, multi-layer model.

---

1. The 'Why'

You now have the pieces: nn.Module for structure (01) and the core layers (02). A working model is just those layers arranged into a sequence, each feeding the next, wrapped so PyTorch tracks every parameter. The classic starting architecture is the multi-layer perceptron (MLP) — a stack of Linear layers with ReLU non-linearities between them. It's the "hello world" of neural networks, and understanding how to build one cleanly transfers directly to every fancier architecture later in the course.

PyTorch gives you two idioms for composing layers, and knowing when to use each is the skill this sub-module builds. nn.Sequential is a quick container that pipes an input straight through a fixed list of layers — perfect when your data just flows in a straight line. A custom nn.Module with an explicit forward method is more verbose but far more flexible: it lets you add branches, skip connections, conditionals, or reuse a layer multiple times. Beginners often reach only for Sequential and hit a wall the moment their architecture isn't a straight line; learning the custom pattern now means you'll never be boxed in.

---

2. Core Concepts

nn.Sequential. A container that chains layers in order and runs them one after another. You get a working model in a few lines, no forward method needed. Great for simple, linear stacks.

Custom nn.Module. Declare layers in __init__, then write forward to control exactly how data moves. Necessary whenever the flow isn't a simple straight line, or when you want readable, named components.

Registering lists of layers. A plain Python list won't register its layers. Use nn.Sequential or nn.ModuleList so the parameters are visible to the optimizer.

Shape discipline. Each layer's output feature size must match the next layer's input size. The output layer's size is dictated by your task (e.g. number of classes for classification, 1 for scalar regression).

---

3. Code in Action

The quick way: nn.Sequential

import torch
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(4, 16),   # input: 4 features -> hidden: 16
    nn.ReLU(),          # non-linearity
    nn.Linear(16, 8),   # hidden: 16 -> hidden: 8
    nn.ReLU(),          # non-linearity
    nn.Linear(8, 3),    # hidden: 8 -> output: 3 (e.g. 3 classes)
)

x = torch.randn(10, 4)  # batch of 10 samples, 4 features each
out = model(x)          # data flows straight through the stack
print(out.shape)        # torch.Size([10, 3])

The flexible way: a custom MLP

import torch
import torch.nn as nn

class MLP(nn.Module):
    def __init__(self, in_dim, hidden, out_dim, p=0.2):
        super().__init__()                     # initialize nn.Module first
        self.fc1 = nn.Linear(in_dim, hidden)   # first linear layer
        self.fc2 = nn.Linear(hidden, out_dim)  # output linear layer
        self.act = nn.ReLU()                   # shared activation
        self.drop = nn.Dropout(p)              # regularization

    def forward(self, x):
        x = self.act(self.fc1(x))   # linear -> ReLU
        x = self.drop(x)            # dropout (active only in train mode)
        return self.fc2(x)          # final linear -> raw scores (logits)

model = MLP(in_dim=4, hidden=16, out_dim=3)
out = model(torch.randn(10, 4))
print(out.shape)        # torch.Size([10, 3])

Confirming the whole model is tracked

# Count total trainable parameters across every registered layer
total = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Trainable parameters: {total}")   # a single number summing all weights + biases

---

4. Common Pitfalls

Pitfall 1 — Layers hidden in a plain list. Writing self.layers = [nn.Linear(4, 8), nn.Linear(8, 3)] silently fails to register the layers, so model.parameters() misses them and they never train. Use nn.Sequential or nn.ModuleList.

Pitfall 2 — Applying a final activation you didn't intend. For classification, most loss functions (like nn.CrossEntropyLoss, Module 05) expect raw logits and apply the softmax internally. Adding a Softmax as your last layer double-applies it and hurts training. Leave the final layer linear unless you have a specific reason.

Pitfall 3 — Mismatched hidden dimensions. When hand-wiring layers, an easy slip is nn.Linear(16, 8) followed by nn.Linear(16, 3) — the 8 and 16 don't line up and the forward pass throws a shape error. Read the dimensions as a chain: each out must equal the next in.

---

5. Further Reading & Watch List

• 📖 Deep Learning with PyTorch, Ch. 6 — composing modules
• 🌐 Learn PyTorch — Building a multi-layer model
• 🛠️ nn.Sequential docs · nn.ModuleList docs
• ▶️ Karpathy — building an MLP in the "makemore" series

---

⬅️ Prev: 02 · Common Layers · ➡️ Next: 04 · Inspecting Models