02 · Common Layers 🧩

4 min read

A network is only as good as the layers it's built from. Three of them — Linear, ReLU, and Dropout — will carry you a very long way.

1. The 'Why'

A neural network learns by transforming its input through a sequence of small, composable operations called layers. Rather than reinventing these each time, PyTorch ships them in torch.nn as ready-made nn.Module subclasses that create and manage their own parameters. Understanding what the most common layers actually do — not just how to type their names — is what lets you design a model on purpose instead of copying one and hoping. Three layers form the backbone of nearly every fully-connected network, and they each play a distinct role.

nn.Linear is the workhorse: it performs the weighted sum (Wx + b) that mixes information across features — this is where the learning lives, in those weights. But stacking linear layers alone is pointless, because a chain of linear operations collapses into a single linear operation; the model could never capture anything curved or complex. That's why we insert a non-linear activation like nn.ReLU between them — it bends the function so the network can approximate rich, non-linear relationships. Finally, nn.Dropout fights overfitting by randomly switching off neurons during training, forcing the network to learn robust, redundant representations instead of memorizing the training set. Master these three and you understand the skeleton of most feed-forward models.

2. Core Concepts

nn.Linear(in_features, out_features). Applies y = xWᵀ + b. It holds a weight matrix and a bias vector as parameters, both tracked by autograd. It maps a vector of in_features numbers to out_features numbers.

Activation functions (nn.ReLU). A non-linearity applied element-wise. ReLU (max(0, x)) is the default choice: cheap, simple, and effective. Without a non-linearity between linear layers, depth buys you nothing. Others you'll meet: Sigmoid, Tanh, GELU.

nn.Dropout(p). During training, randomly zeros each element with probability p, which regularizes the model. During evaluation it does nothing (a pass-through). This train/eval difference is why model.train() / model.eval() matter — covered in sub-module 04.

Module vs. functional form. Many layers exist both as a class (nn.ReLU()) and a function (F.relu(x)). Use the class form for things with parameters or state; either works for stateless ops like ReLU.

3. Code in Action

`nn.Linear` — the weighted sum

import torch
import torch.nn as nn

layer = nn.Linear(in_features=3, out_features=2)  # maps 3 numbers -> 2 numbers
x = torch.randn(4, 3)          # a batch of 4 samples, each with 3 features
out = layer(x)                 # apply Wx + b to every sample
print(out.shape)               # torch.Size([4, 2]) -> batch preserved, features remapped

# The learnable parameters live inside the layer:
print(layer.weight.shape)      # torch.Size([2, 3])
print(layer.bias.shape)        # torch.Size([2])

`nn.ReLU` — adding non-linearity

import torch
import torch.nn as nn

relu = nn.ReLU()
x = torch.tensor([-2.0, -0.5, 0.0, 1.5])
print(relu(x))                 # tensor([0., 0., 0., 1.5]) -> negatives clamped to 0

# Equivalent functional form:
import torch.nn.functional as F
print(F.relu(x))               # same result, no layer object needed

`nn.Dropout` — regularization that respects mode

import torch
import torch.nn as nn

drop = nn.Dropout(p=0.5)       # zero ~50% of elements during training
x = torch.ones(1, 8)

drop.train()                   # training mode: dropout is ACTIVE
print(drop(x))                 # ~half the entries become 0; survivors scaled up

drop.eval()                    # evaluation mode: dropout is a PASS-THROUGH
print(drop(x))                 # all ones — no elements dropped

4. Common Pitfalls

Pitfall 1 — Stacking linear layers with no activation. nn.Linear followed directly by nn.Linear is mathematically equivalent to a single linear layer — the depth is wasted. Always place a non-linearity (like nn.ReLU) between them so the network can model complex functions.

Pitfall 2 — Mismatched in_features. A layer's in_features must equal the size of the incoming feature dimension, or you get a shape-mismatch error at the first forward pass. When chaining layers, the out_features of one must equal the in_features of the next. Trace the shapes on paper.

Pitfall 3 — Forgetting model.eval() and expecting stable outputs. Because nn.Dropout (and BatchNorm) behave differently in training vs. evaluation, running inference while still in training mode gives randomly varying, worse predictions. Always switch to model.eval() before evaluating — detailed in sub-module 04.

5. Further Reading & Watch List

⬅️ Prev: 01 · The nn.Module Basics · ➡️ Next: 03 · Building a Network