Neural Networks & Deep Learning

Module Completed: November 2021

---

🎯 Module Overview

Neural networks are the foundation of modern AI — from image recognition to natural language understanding. This module builds up from the biological neuron analogy through to multi-layer deep networks, covering the mathematics of forward and backward propagation, activation functions, loss functions, optimisers, regularisation, and practical model building in TensorFlow/Keras.

---

📖 Topics Covered

Biological vs Artificial Neurons

The biological neuron receives signals via dendrites, processes them in the cell body, and fires an output signal through the axon. The artificial neuron mirrors this:

• Inputs (x) → multiplied by Weights (w) → summed → Bias (b) added → passed through Activation Function → output
• A single neuron = a linear classifier (like logistic regression)
• Stacking neurons in layers enables learning non-linear decision boundaries

Neural Network Architecture

• Input Layer — one node per feature; no computation, just passes inputs
• Hidden Layers — where representation learning happens; each layer transforms the previous layer's output
• Output Layer — 1 neuron for regression; 1 neuron (sigmoid) for binary classification; N neurons (softmax) for multi-class
• Depth (number of layers) vs Width (neurons per layer) — deeper networks capture more complex patterns

Forward Propagation

Data flows left to right: input → weighted sum → activation → next layer. For each layer l:

z = W·x + b (linear combination) a = f(z) (activation function applied)

Activation Functions

Transform the weighted sum into the layer's output; introduces non-linearity (without it, a deep network collapses to a linear model):

Function	Formula	Range	Use Case
Sigmoid	1/(1+e⁻ˣ)	(0, 1)	Binary output layer
Tanh	(eˣ−e⁻ˣ)/(eˣ+e⁻ˣ)	(−1, 1)	Hidden layers (zero-centred)
ReLU	max(0, x)	[0, ∞)	Default for hidden layers; avoids vanishing gradient
Leaky ReLU	max(0.01x, x)	(−∞, ∞)	Fixes dying ReLU problem
Softmax	eˣⁱ / Σeˣʲ	(0,1), sums to 1	Multi-class output layer

Vanishing Gradient Problem: Sigmoid and Tanh saturate at extremes → gradients become nearly zero → early layers learn very slowly. ReLU largely solves this.

Loss Functions

Measure how wrong the model's predictions are; the quantity optimised during training:

• MSE (Mean Squared Error) — regression; penalises large errors heavily
• MAE (Mean Absolute Error) — regression; more robust to outliers
• Binary Crossentropy — binary classification; measures divergence between predicted probability and true label
• Categorical Crossentropy — multi-class classification with one-hot labels
• Sparse Categorical Crossentropy — same as above but takes integer class labels (no one-hot needed)

Backpropagation

The algorithm that computes gradients of the loss with respect to every weight in the network using the chain rule of calculus. Works backwards from output to input:

1. Compute loss on forward pass
2. Compute gradient of loss w.r.t. output layer weights
3. Propagate gradients backwards through each layer using chain rule
4. Update all weights using the gradients

Gradient Descent & Optimisers

Update rule: W = W − α × ∂L/∂W (α = learning rate)

Optimiser	Key Idea	When to Use
SGD	Update after each sample (or mini-batch)	Simple baselines
Momentum	Accumulates past gradients to smooth updates	Avoids oscillation
RMSProp	Adaptive learning rate per parameter	RNNs, non-stationary objectives
Adam	Combines Momentum + RMSProp; most popular	Default choice for most problems
Adagrad	Larger updates for rare features	Sparse data (NLP)

Learning Rate: Too high → overshoots minimum; too low → slow convergence. Learning rate schedulers (step decay, cosine annealing) reduce LR during training.

Batch Size & Epochs

• Epoch — one full pass through the entire training dataset
• Batch Size — number of samples processed before each weight update; trade-off between gradient quality and training speed
  • Small batch (e.g., 10) — noisier gradients, acts as regulariser, slower per epoch
  • Large batch (e.g., 256) — smoother gradients, faster per epoch, may generalise worse
• Validation Split — hold out a portion of training data to monitor overfitting during training

Regularisation in Neural Networks

Prevent overfitting in deep networks:

• Dropout — randomly sets a fraction of neuron outputs to zero during training; forces redundant representations; Dropout(rate=0.5) drops 50% of neurons
• Batch Normalisation — normalises layer inputs to zero mean/unit variance before activation; stabilises training, allows higher learning rates; acts as mild regulariser
• L1/L2 Weight Regularisation — adds penalty term to loss for large weights; kernel_regularizer=l2(0.001)
• Early Stopping — stop training when validation loss stops improving; use EarlyStopping(patience=5) callback

Model Building in TensorFlow/Keras

• Sequential API — linear stack of layers; model.add(Dense(units, activation))
• Functional API — for complex architectures with branching, skip connections
• model.compile(optimizer, loss, metrics) — sets up training configuration
• model.fit(X, y, epochs, batch_size, validation_split) — runs training
• model.save() / load_model() — save/load entire model including weights and architecture (preferred over pickle for Keras models)

Hyperparameter Tuning for Neural Networks

Key hyperparameters to tune:

• Number of hidden layers and neurons per layer
• Activation function
• Learning rate and optimiser
• Batch size and number of epochs
• Dropout rate

Introduction to CNN (Basics)

• Convolutional Layer — applies learnable filters across the image to detect local features (edges, textures); parameters shared across spatial locations
• Pooling Layer — reduces spatial dimensions (Max Pooling, Average Pooling); provides translation invariance
• Flatten — converts 2D feature maps to 1D vector before Dense layers
• CNNs drastically reduce parameters vs fully-connected networks on image data

Data Augmentation & Preprocessing for Images

• Resizing, cropping — reduce unwanted information in frame
• Normalising pixel values from (0, 255) → (0, 1) or (-1, 1)
• Applying image filters: Sobel (edge detection), Gaussian (smoothing), Laplace
• One-hot encoding target class for multi-class image classification
• Reshaping to add channel dimension: (height, width) → (height, width, channels)

---

🏆 Project — Introduction to Neural Networks & Deep Learning

Submitted: 14 November 2021

---

Part 1 — Signal Quality Regression (MLP Regressor)

Domain: Electronics & Telecommunication

Business Problem: Predict the quality/strength of a signal from measurable signal parameters.

Dataset: Part-1 - Signal.csv — 1,599 records × 12 features (Parameter 1–11 + Signal_Quality target)

EDA Findings:

• No null values; all features are float64
• Wide range of scales (Parameter 7: 46–289; Parameter 3: 0–1) → normalisation mandatory
• Moderate correlation between some parameters (Parameter 9 vs Parameter 1: −0.68; Parameter 8 vs Parameter 1: +0.68) — below 0.8, no removal needed
• Outliers present — retained (neural networks are robust to outliers)

Model Architecture:

• Batch Normalisation layer (before hidden layers)
• Hidden Layer 1: 10 neurons, ReLU
• Hidden Layer 2: 10 neurons, ReLU
• Output Layer: 1 neuron (regression, no activation)
• Optimiser: SGD | Loss: MSE

Training:

• Best config: batch_size=10, epochs=30
• 80/20 train/validation split within training data
• Both training loss and validation loss converged to ~0.4
• Slight validation loss increase after epoch 25 → best checkpoint at epoch 25
• Model saved using Keras model.save() (not pickle)

---

Part 2 — Street View House Number (SVHN) Digit Recognition (CNN)

Domain: Autonomous Vehicles / Map Making

Business Problem: Recognise single digits in street-level photographs (Google Street View House Numbers dataset) — a key component of automated map making.

Dataset: SVHN dataset in HDF5 format — training, validation, and test splits; grayscale images of house number digits (0–9)

Challenge: Images often contain more than one digit in frame — misleading for single-digit classification.

Data Preprocessing Pipeline:

1. Cropping — crop images to reduce multi-digit ambiguity (accuracy improved by ~10%)
2. Filtering — tested Sobel, Correlate, Gaussian Gradient Magnitude, Gaussian Filter, Laplace; Sobel filter performed best (sharpens digit edges)
3. Normalisation — pixel values scaled from (0, 255) → (0, 1)
4. Reshape — added channel dimension: (H, W) → (H, W, 1) (grayscale single channel)
5. One-hot encoding — 10 target classes (digits 0–9)

Model Architecture:

• Flatten layer (image array → 1D)
• Hidden Layer 1: 256 neurons, ReLU
• Hidden Layer 2: 128 neurons, ReLU
• Output Layer: 10 neurons, Softmax
• Optimiser: SGD | Loss: Categorical Crossentropy

Training Results:

Config	Validation Accuracy
No preprocessing	~68%
Cropping only	~70%
Cropping + Sobel filter	~83%
batch=100, epochs=30 (final)	~83%

Key Observation: Preprocessing improved accuracy by ~15%; model converged and plateaued after epoch 20.

---

🔑 Key Learnings

1. Without non-linear activations, deep networks = linear models — activation functions are what give depth its power
2. ReLU is the default activation for hidden layers — avoids vanishing gradient; use Sigmoid/Softmax only at the output
3. Batch Normalisation before activations speeds up training — normalises layer inputs; allows higher learning rates; acts as regulariser
4. Small batch size = implicit regularisation — noisy gradient updates prevent the model from memorising training data
5. Monitor train AND validation loss — training loss alone is misleading; a growing gap between train/val loss signals overfitting
6. Use model.save() not pickle for Keras models — saves full architecture + weights + training config; pickle only saves Python objects
7. Data preprocessing is often more impactful than model architecture — the Sobel filter + cropping improved SVHN accuracy by ~15%; a bigger network without preprocessing only reached 68%
8. Epoch patience matters — validation loss can temporarily increase before continuing to fall; Early Stopping with patience=5 prevents premature stopping

---

🛠️ Tools & Libraries

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import h5py

import tensorflow as tf
from tensorflow import keras
from keras.models import Sequential, load_model
from keras.layers import (Dense, Dropout, BatchNormalization,
                          Flatten, Conv2D, MaxPooling2D)
from keras.optimizers import SGD, Adam
from keras.utils import to_categorical
from keras.callbacks import EarlyStopping, ModelCheckpoint

from scipy.ndimage import sobel, gaussian_filter, laplace
from sklearn.preprocessing import MinMaxScaler