Computer Vision (Object Detection & Segmentation)

Module Completed: January 2022

---

🎯 Module Overview

This module pushes beyond image classification into two advanced computer vision domains: semantic segmentation (identifying which pixels belong to which object) and face recognition (one-shot / few-shot learning). It covers U-Net architecture for segmentation, VGGFace embeddings for identity matching, and metric learning concepts.

---

📖 Topics Covered

Object Detection — Conceptual Overview

Object detection predicts both class and location (bounding box) for all objects in an image. Slides covered:

Region-Based Methods:

• R-CNN — extract ~2,000 region proposals → CNN feature extraction per region → SVM classify; slow (forward pass per region)
• Fast R-CNN — run CNN once over whole image → project regions onto feature map → classify; much faster
• Faster R-CNN — Region Proposal Network (RPN) built into the CNN; end-to-end trainable; real-time feasible

Single-Stage Detectors:

• YOLO (You Only Look Once) — divides image into grid; each cell predicts bounding boxes + class probabilities in a single forward pass; real-time detection (30+ FPS)
  • Anchor boxes — predefined aspect ratios per grid cell
  • objectness score × class probability = final confidence
• SSD (Single Shot Detector) — multi-scale feature maps for detecting objects at different sizes; fast and accurate

Key Concepts:

• IoU (Intersection over Union) — overlap between predicted and ground-truth box; IoU > 0.5 typically = correct detection
• Non-Maximum Suppression (NMS) — removes duplicate detections; keeps highest-confidence box per object
• Anchor Boxes — prior bounding box shapes that the model learns offsets from

Semantic Segmentation

Assigns a class label to every pixel in the image (not just bounding boxes):

• FCN (Fully Convolutional Network) — replaces Dense layers with Conv layers; outputs spatial prediction map
• U-Net — encoder-decoder architecture with skip connections between corresponding encoder/decoder layers
  • Encoder — successive Conv+Pool blocks; downsamples, extracts features
  • Decoder — successive UpSampling+Conv blocks; reconstructs spatial resolution
  • Skip connections — directly connect encoder layers to decoder at same resolution; recovers fine spatial detail lost during downsampling
  • Originally designed for biomedical image segmentation; excellent with small datasets
• Dice Coefficient — overlap metric for segmentation masks: 2×|A∩B| / (|A|+|B|); ranges 0–1; higher = better mask overlap. More robust than pixel accuracy for imbalanced foreground/background

Instance Segmentation

• Mask R-CNN — extends Faster R-CNN with a third branch predicting a pixel-level mask for each detected object; handles multiple instances of same class separately

Face Recognition & Metric Learning

Different paradigm from classification — rather than classifying into fixed classes, learn an embedding space where same-person images are close and different-person images are far:

• VGGFace — deep CNN trained on large face datasets; outputs a 2,622-dimensional embedding vector per face image
• Embedding — a compact, dense vector representation that encodes identity information; (224, 224, 3) image → (2622,) embedding
• Distance Metrics — measure similarity between embeddings:
  • Euclidean distance — L2 norm between embedding vectors
  • Cosine similarity — angle between embedding vectors
• One-Shot / Few-Shot Learning — recognise a person from very few (even 1) example images; enabled by embedding-based approaches + distance thresholding
• Siamese Networks — twin networks sharing the same weights; trained with contrastive or triplet loss to push same-class embeddings together and different-class apart

Triplet Loss & Contrastive Loss

• Contrastive Loss — for pairs: minimise distance for same-class pairs, maximise for different-class pairs
• Triplet Loss — for triplets (Anchor, Positive, Negative): Loss = max(0, d(A,P) - d(A,N) + margin) — ensures anchor is closer to positive than negative by at least margin

---

🏆 Project — Advanced Computer Vision (CV-2)

Submitted: 02 January 2022

---

Part 1 — Face Mask Detection / Segmentation (U-Net)

Domain: Entertainment (movie scene cast identification)

Business Problem: When a user pauses a movie, the app should detect faces in the frame and overlay a mask to enable cast lookup.

Dataset: 409 training images + corresponding face masks

Approach:

1. Created custom mask images: background = white, face mask overlay = red (after multiple trials — best Dice Coefficient achieved)
2. Built a custom Dice Coefficient metric and corresponding Dice Loss function
3. Designed a U-Net architecture:
   • Encoder: Conv2D → MaxPool blocks (progressively smaller feature maps)
   • Bottleneck: deepest feature representation
   • Decoder: UpSampling → Conv2D blocks with skip connections from encoder
   • Output: sigmoid activation per pixel (binary mask)
4. Train/test split: only 1% held for evaluation (dataset too small — every image needed for training)

Results:

• Good Dice Coefficient on training data
• Some evaluation images missed (expected — only 409 total training images)
• Prediction image: successfully masked faces of both persons in scene
• Key challenge: Minimum training data and variable image quality limited generalisation

---

Part 2 — Face Recognition using VGGFace + SVM (One-Shot Learning)

Domain: Security / Identity Verification

Business Problem: Identify actors/persons by face from images using a face recognition model.

Dataset: Face Aligned Dataset from Pinterest — 10,770 images, 100 people
Directory structure: PINS/pins_<person_name>/ → images per person

Pipeline:

1. Load pre-trained VGGFace model — deep CNN (224×224×3 input) trained on large face datasets
2. Generate embedding vectors — each face image → (2622,) embedding vector via VGGFace feature extractor (include_top=False + GlobalAveragePooling)
3. Label Encoding — class names (person names) → integer labels
4. StandardScaler — normalise embedding dimensions
5. PCA — dimensionality reduction on 2,622-dimensional embeddings for faster SVM training
6. SVM Classifier — trained on PCA-reduced embeddings to map embedding → person identity

Results:

• SVM accuracy on test set: 95%
• Both prediction images correctly identified

Why this works — Embedding-based face recognition:

• VGGFace has already learned rich face-discriminative features from millions of face images
• The 2,622-dim embedding encodes identity; similar faces are close in this space
• SVM simply draws decision boundaries in embedding space — much easier than training from scratch

---

🔑 Key Learnings

1. Segmentation ≠ classification — pixel-level prediction requires architectures (U-Net, FCN) that preserve and recover spatial resolution; Dice Coefficient is the right metric, not accuracy
2. Skip connections in U-Net are crucial — without them, the decoder cannot recover fine spatial details lost during downsampling; masks are blurry
3. Dice Loss > Binary Crossentropy for segmentation — highly imbalanced foreground/background makes pixel-accuracy misleading; Dice directly measures mask overlap
4. For face recognition, use embeddings not classification — fixed-class classification can't handle new persons without retraining; embedding + distance threshold generalises to unseen identities
5. Pre-trained VGGFace >> training from scratch — 10k images is far too few to learn facial features from scratch; the pre-trained model already knows faces deeply
6. YOLO for real-time, Faster R-CNN for accuracy — choose based on latency requirement; YOLO runs 30+ FPS, Faster R-CNN is more accurate but slower
7. IoU and NMS are critical post-processing steps — raw detection outputs have duplicates; NMS selects the best box per object based on confidence + overlap
8. Mask colour choice affects Dice Coefficient — white background + red mask gave best segmentation performance vs other colour combinations

---

🛠️ Tools & Libraries

import numpy as np
import matplotlib.pyplot as plt
import cv2
from PIL import Image

import tensorflow as tf
from tensorflow import keras
from keras.models import Model, Sequential, load_model
from keras.layers import (Conv2D, MaxPooling2D, UpSampling2D, Dense,
                           Dropout, Flatten, Input, concatenate,
                           GlobalAveragePooling2D)
from keras.optimizers import Adam
from keras.applications import VGG16

# VGGFace (keras-vggface)
from keras_vggface.vggface import VGGFace

from sklearn.svm import SVC
from sklearn.preprocessing import LabelEncoder, StandardScaler
from sklearn.decomposition import PCA
from sklearn.metrics import accuracy_score