Recommendation Systems

Module Completed: October 2021

---

🎯 Module Overview

Recommendation systems power product discovery at companies like Netflix, Amazon, and Spotify. They predict what a user is likely to enjoy based on past behaviour and the behaviour of similar users or items. This module covers the full spectrum — from simple popularity-based models through to matrix factorisation — along with evaluation metrics and real-world trade-offs.

---

📖 Topics Covered

Types of Recommendation Systems

Type	How It Works	When to Use
Popularity-Based	Recommends globally most-rated/trending items	New users (cold start), no personal history available
Content-Based	Matches items to user profile based on item attributes	Rich item metadata available; user history limited
Collaborative Filtering	Finds users/items with similar rating patterns	Sufficient ratings history exists for users and items
Hybrid	Combines 2+ of the above	Production systems needing both coverage and personalisation

Popularity-Based Recommendation

The simplest approach — rank items by aggregate popularity and recommend the top N. Key insight: raw average rating is misleading for items with few ratings. Uses the IMDB Weighted Rating formula instead:

W = (R×v + C×m) / (v + m)

• R = average rating of the item
• v = number of votes for the item
• C = mean vote across all items
• m = minimum votes threshold to be listed

This Bayesian-style weighting pulls items with few votes towards the global mean, ensuring well-rated but rarely-reviewed items don't dominate.

Limitation: Same recommendations for all users — not personalised. Best used for new users with no history (cold start).

Similarity Measures

The mathematical foundation of collaborative filtering:

• Cosine Similarity — angle between two rating vectors; ignores magnitude, captures direction of preference; most common in RS
• Pearson Correlation — measures linear relationship; normalises for rating scale bias (users who consistently rate high/low)
• Euclidean Distance — raw distance; sensitive to scale differences
• Jaccard Similarity — used for binary interaction data (clicked / not clicked)

User-User Collaborative Filtering

Finds users similar to the target user, then recommends items those similar users rated highly but the target user hasn't seen yet.

• Build a user-item rating matrix
• Compute pairwise user similarity (cosine or Pearson)
• For each unseen item, predict rating as a weighted average of similar users' ratings
• KNNWithMeans in Surprise: subtracts each user's mean rating before computing similarity (corrects for rating bias)
• Limitation: Scales poorly with large user bases (O(n²) similarity computation)

Item-Item Collaborative Filtering

Computes item-item similarity instead of user-user. For a target user, find items similar to those they already rated positively.

• More stable than user-user (item preferences change slower than user tastes)
• Used by Amazon ("Customers who bought X also bought Y")
• KNNWithMeans(sim_options={'user_based': False}) in Surprise

Matrix Factorisation — SVD

Decomposes the sparse user-item rating matrix into latent factor matrices:

R ≈ U × Σ × Vᵀ

• U — user latent factor matrix (users × k factors)
• Σ — singular values (importance of each factor)
• V — item latent factor matrix (items × k factors)
• Latent factors capture hidden concepts (e.g., "action movie fan", "budget-conscious buyer") without explicit labels
• Handles sparsity — only rated entries are used for training
• SVD in Surprise library uses SGD (Stochastic Gradient Descent) to learn factors
• Outperforms KNN-based methods for large, sparse matrices
• SVD++ — extends SVD with implicit feedback (viewed, clicked) in addition to explicit ratings

Evaluation Metrics

• RMSE (Root Mean Squared Error) — primary metric; penalises large prediction errors more heavily; lower = better
• MAE (Mean Absolute Error) — average absolute difference; more interpretable
• Precision@K — of the top K recommendations, how many are relevant
• Recall@K — of all relevant items, how many appear in the top K
• Cross-validation with cross_validate from Surprise — gives mean RMSE/MAE across folds

The Cold Start Problem

• New User Cold Start — no rating history → can't personalise → use popularity-based or demographic-based fallback
• New Item Cold Start — new product with no ratings → can't use CF → use content-based or push via promotional slots
• Solutions: Ask users for preferences at onboarding; use hybrid models; exploit implicit signals (clicks, browse history)

Association Rules & Apriori (Market Basket Analysis)

Related technique for "frequently bought together" recommendations:

• Support — how often itemset appears in transactions
• Confidence — P(B | A) — given A was bought, probability B was also bought
• Lift — confidence / expected confidence; lift > 1 means the association is stronger than random
• mlxtend library: apriori() + association_rules()

Surprise Library Algorithms (Full Taxonomy)

• Basic: NormalPredictor, BaselineOnly
• KNN-Based: KNNBasic, KNNWithMeans, KNNWithZScore, KNNBaseline
• Matrix Factorisation: SVD, SVD++, NMF
• Other: Slope One, Co-Clustering

---

🏆 Project — Mobile Phone Recommendation System

Submitted: 03 October 2021 Domain: Smartphone / Electronics

---

Problem Statement

Build a recommendation system to suggest mobile phones to users using:

1. A popularity-based model (for all users)
2. A collaborative filtering model (personalised)

---

Dataset

• Source: 6 CSV files merged → 1,415,133 raw records
• Features: author (user), product (phone name), score (rating), score_max, country, date, domain, language, extract, source
• Final working dataset: 41,892 records × 3 columns (author, product, score)

---

Data Preparation

1. Merge 6 CSVs into one dataframe
2. Round score to nearest integer
3. Drop nulls in author, score, product — imputing rating data would be misleading
4. Remove duplicates — 4,677 duplicates dropped; further deduplication after column reduction
5. Sample 1,000,000 records (random_state=612) from 1.28M cleaned records
6. Filter — keep only products with > 50 ratings AND users who gave > 50 ratings → 41,892 records

---

Key EDA Findings

• Most-rated product: OnePlus 3 (Graphite, 64 GB)
• User with most reviews: Amazon Customer (14,582 reviews)
• Raw average rating is misleading — a product with score=10 and 1 review vs score=8.2 with 168 reviews; IMDB weighted formula used to resolve

---

Model 1 — Popularity-Based

Applied IMDB Weighted Rating formula (m = minimum rating count threshold, C = global mean rating).

Top 5 Recommendations (same for all users):

1. Apple iPhone 5s 16GB 2–5: Other high weighted-rating phones

Use case: Shown to new users with no rating history

---

Model 2 — Collaborative Filtering (Surprise Library)

Three algorithms trained and compared:

Algorithm	Type	RMSE
SVD	Matrix Factorisation	Lowest (best)
KNNWithMeans (User-based)	User-User CF	Slightly higher
KNNWithMeans (Item-based)	Item-Item CF	Slightly higher

SVD selected — considers both user and item latent factors simultaneously; better RMSE than KNN-based methods.

Cross-validation: Applied 5-fold CV to all 3 algorithms — marginal RMSE improvement; confirms models are not overfitting.

Inference example: For user "Rahul" — SVD algorithm outputs top 5 personalised phone recommendations based on his rating history and latent factor similarity to other users.

---

Business Scenario Mapping

Scenario	Recommended System
No user history available	Popularity-based
New user just signed up	Popularity-based → transition to CF as ratings accumulate
Existing user with rating history	Collaborative Filtering (SVD)
Rich product attributes (specs, genre)	Content-Based Filtering
Production (best of all)	Hybrid (CF + Content-Based)

---

🔑 Key Learnings

1. Raw average rating is not popularity — always use a weighted formula (IMDB-style) that accounts for number of votes; a 10/10 from 1 user ≠ 8.5/10 from 500 users
2. Filter sparse users and items — users with < 50 ratings and items with < 50 ratings add noise; filtering creates a denser, more reliable matrix
3. SVD > KNN for large sparse matrices — matrix factorisation captures latent structure; KNN fails as sparsity increases
4. Never impute missing ratings — in RS, a missing rating means "unseen", not "zero preference"; imputing would corrupt the signal
5. Cold start requires a fallback — popularity-based or demographic-based model must run alongside CF for new users/items
6. Cross-validation in Surprise — use cross_validate() for robust RMSE estimates across folds; single train/test is insufficient
7. Item-item CF is more stable than user-user — item preferences change more slowly than user behaviour patterns; scales better too
8. Lift > 1 ≠ useful recommendation — a high-lift association rule may still have very low support; both metrics needed for market basket analysis

---

🛠️ Tools & Libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Collaborative Filtering
from surprise import SVD, KNNWithMeans, Dataset, Reader, accuracy
from surprise.model_selection import train_test_split, cross_validate

# Market Basket / Apriori
from mlxtend.frequent_patterns import apriori, association_rules
from mlxtend.preprocessing import TransactionEncoder

# Similarity
from sklearn.metrics.pairwise import cosine_similarity