Background
Back to Articles/PGP in Artificial Intelligence & Machine Learning
Sections
IntroductionApplied_StatisticsSupervised LearningEnsemble TechniquesUnsupervised LearningFeature Engineering & Model TuningRecommendation SystemsNeural Networks & Deep LearningComputer Vision (CNN & Transfer Learning)Computer Vision (Object Detection & Segmentation)Statistical NLPSequential NLPCapstone Project - AI & ML

Statistical NLP

7 min read

Module Completed: January 2022

🎯 Module Overview

Natural Language Processing (NLP) enables machines to read, understand, and generate human language. This module covers the statistical foundations of NLP — from raw text cleaning through to vector representations and classical ML classifiers for text. The project tackles a real multi-label blog authorship classification problem spanning 600,000+ posts.

📖 Topics Covered

Text Preprocessing Pipeline

Before any model can consume text, it must be cleaned and normalised. Standard steps:

  1. Language Detection — filter to target language only; fasttext library is faster and more accurate than langdetect for large corpora
  2. Special character & number removal — strip punctuation, digits, URLs using regex
  3. Lowercasing — normalise case so "The" and "the" are treated identically
  4. Stop word removal — remove high-frequency, low-information words (the, is, at, which); nltk.corpus.stopwords
  5. Tokenisation — split text into individual tokens (words or subwords)
  6. Stemming — reduce words to root form by rule-based suffix stripping (e.g., "running" → "run"); fast but crude; nltk.PorterStemmer
  7. Lemmatisation — reduce words to dictionary base form using vocabulary lookup (e.g., "better" → "good"); slower but more accurate; nltk.WordNetLemmatizer
  8. N-grams — sequences of N consecutive tokens; bigrams (N=2) capture "New York" as a single feature

Bag of Words (BoW)

Represents text as a vector of word frequency counts — ignores word order, purely frequency-based:

  • Each unique word = one dimension in the vocabulary
  • Each document = a sparse vector of counts
  • CountVectorizer(ngram_range=(1,2), analyzer='word') — includes unigrams and bigrams
  • Fast, interpretable, but ignores context and treats all words as equally important

TF-IDF (Term Frequency — Inverse Document Frequency)

Improves on BoW by weighting terms by their importance:

TF-IDF(t, d) = TF(t, d) × IDF(t)

  • TF = frequency of term t in document d
  • IDF = log(N / df(t)) where N = total docs, df(t) = docs containing t
  • Common words that appear in many documents get lower IDF (lower weight)
  • Rare, discriminative words get higher IDF (higher weight)
  • Result: more meaningful feature representation than raw counts
  • TfidfVectorizer(ngram_range=(1,1), analyzer='word')

HashingVectorizer

Applies a hash function to map tokens to feature indices — does not build a vocabulary dictionary:

  • Fixed-size output regardless of vocabulary size → memory efficient for very large corpora
  • Cannot invert (no inverse_transform) — useful for streaming/online learning
  • May have hash collisions (different words mapped to same index)
  • HashingVectorizer(ngram_range=(1,1))

Multi-Label Classification

A sample can belong to multiple classes simultaneously (e.g., a blog post has gender + age + topic + sign labels). Key differences from multi-class:

  • MultiLabelBinarizer — encodes multiple labels per instance into a binary matrix
  • OneVsRestClassifier — trains one binary classifier per label; each independently predicts if the label applies
  • Evaluation metrics: accuracy alone is misleading; use f1_score, average_precision_score, average_recall_score — these account for partial matches across labels

Classifiers for Text

Text features (after vectorisation) fed into classical ML models:

Model Strength for Text
Logistic Regression Fast, interpretable baseline; struggles at very high dimensions
SVC Effective for high-dimensional sparse features; slow on large corpora
Random Forest Ensemble; feature importance available; handles non-linearity
XGBoost Best performer on large multi-feature text tasks; gradient boosting captures complex patterns; feature importance built-in

Key insight: For large corpora with many features, ensemble methods (Random Forest, XGBoost) outperform linear models. XGBoost's gradient boosting in feature space gives it an edge over Random Forest's hyperparameter-based optimisation.

Vectoriser Comparison

  • CountVectorizer — simple frequency; fast; can be dominated by high-frequency words
  • TfidfVectorizer — downweights common words; generally performs better in practice
  • HashingVectorizer — memory efficient; no vocabulary; collision risk

Word Embeddings — Word2Vec

Dense vector representations where semantically similar words are close in vector space:

  • CBOW (Continuous Bag of Words) — predict centre word from context window
  • Skip-gram — predict context words from centre word; better for rare words
  • Captures semantic relationships: king - man + woman ≈ queen
  • Vocabulary: fixed; unknown words (OOV) cannot be represented
  • gensim.models.Word2Vec

Regular Expressions for NLP

Pattern matching for text cleaning and extraction:

  • . (any char), * (0+), + (1+), ? (0 or 1), ^ (start), $ (end)
  • \d (digit), \w (word char), \s (whitespace), \b (word boundary)
  • Character classes: [a-z], [^aeiou] (negation)
  • Capture groups: (pattern) — extract matched substrings
  • re.sub(pattern, replacement, text) — most common for text cleaning

🏆 Project — Statistical NLP: Blog Authorship Classification (NLP-1)

Submitted: 30 January 2022 Domain: Digital Content Management

Problem Statement

Multi-label classification of blog posts to predict the combined author attributes (gender + age + topic + astrological sign) from blog text.

Dataset

  • Blog Authorship Corpus — 681,288 posts from 19,320 bloggers (blogger.com, August 2004)
  • Features: text (raw blog post), gender, age, topic, sign
  • Age groups: teens (13–17), twenties (23–27), thirties (33–47); equal male/female per group
  • 40 unique topics, 12 unique astrological signs

Data Preparation

  1. Language detection with fasttext — keep only English posts
  2. Text cleaning: remove special chars, numbers, lowercase, stop words, extra whitespace → clean_data column
  3. Combined label: gender + age + topic + sign concatenated → labels target column
  4. MultiLabelBinarizer on labels + OneVsRestClassifier for multi-label setup

Experiments — Vectoriser × Classifier Grid

All combinations of 3 vectorisers × 4 classifiers evaluated:

Vectoriser Classifiers Tested
CountVectorizer (ngram 1–2) Random Forest, SVC, Logistic Regression, XGBoost
TfidfVectorizer (ngram 1–1) Random Forest, SVC, Logistic Regression, XGBoost
HashingVectorizer (ngram 1–1) Random Forest, SVC, Logistic Regression, XGBoost

Results:

  • XGBClassifier performed best across all vectorisers — highest accuracy and F1-score
  • TF-IDF performed marginally better than CountVectorizer overall
  • HashingVectorizer — comparable performance, better memory profile

Hyperparameter Tuning

GridSearchCV applied to XGBClassifier with CountVectorizer and TfidfVectorizer — marginal improvement; model already close to performance ceiling for the given data size/sample.

Key Conclusions

  • Best Model: XGBClassifier + TF-IDF
  • Best Metrics: F1-score and average precision/recall (not raw accuracy — multi-label problem)
  • Why XGBoost: Ensemble gradient boosting in feature space; automatic feature importance; handles large sparse text features better than SVC or LogisticRegression

🔑 Key Learnings

  1. TF-IDF > BoW in practice — downweighting common words gives more discriminative features; but the gap narrows with n-grams
  2. Multi-label ≠ multi-class — one sample can have multiple labels simultaneously; requires MultiLabelBinarizer + OneVsRestClassifier; accuracy alone is misleading
  3. fasttext for language detection — significantly more accurate and faster than rule-based or other library alternatives for large corpora
  4. Stop word removal is a default step — but for some tasks (sentiment analysis), stop words like "not" carry critical meaning; remove carefully
  5. Stemming vs Lemmatisation trade-off — stemming is faster (rule-based), lemmatisation is more accurate (vocabulary-based); use lemmatisation when precision matters
  6. XGBoost for large text features — ensemble methods scale better to high-dimensional sparse feature spaces than linear classifiers
  7. HashingVectorizer for streaming — when vocabulary is too large for memory, hashing maps tokens to fixed-size space without building a dictionary

🛠️ Tools & Libraries

import re
import nltk
import fasttext
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from wordcloud import WordCloud

from sklearn.feature_extraction.text import (CountVectorizer,
                                               TfidfVectorizer,
                                               HashingVectorizer)
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import (f1_score, average_precision_score,
                              classification_report)

from xgboost import XGBClassifier
import gensim
from gensim.models import Word2Vec