Sequential NLP

Module Completed: February 2022

---

🎯 Module Overview

Sequential models process text as ordered sequences, capturing contextual dependencies that statistical approaches miss. This module covers Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), Bidirectional LSTMs, and the attention mechanism that underpins modern transformers. The project applies these to sentiment analysis and sarcasm detection.

---

📖 Topics Covered

Why Sequential Models?

Bag-of-Words and TF-IDF lose word order — "good, not bad" and "bad, not good" have identical BoW representations. Sequential models process words in order, maintaining a hidden state that carries context from earlier positions:

• "I loved it" → context built left to right
• "I didn't love it" → the "didn't" modifies what follows — only captured with sequence awareness

Recurrent Neural Networks (RNN)

An RNN processes input tokens one at a time, maintaining a hidden state h:

hₜ = tanh(Wₓ·xₜ + Wₕ·hₜ₋₁ + b)

The hidden state acts as a "memory" passing information from step to step. One set of weights shared across all time steps (parameter efficiency).

Problems with Vanilla RNN:

• Vanishing Gradient — gradients shrink exponentially as they backpropagate through many time steps; early tokens are "forgotten"
• Exploding Gradient — gradients grow unboundedly; clipped with gradient_clipping
• Struggles with long-range dependencies (e.g., subject-verb agreement across a long sentence)

Long Short-Term Memory (LSTM)

LSTM introduces a cell state (Cₜ) alongside the hidden state, with three learned gates that control information flow:

Gate	Formula	Role
Forget Gate	σ(Wf·[hₜ₋₁, xₜ] + bf)	Decides what to erase from cell state
Input Gate	σ(Wi·[hₜ₋₁, xₜ] + bi)	Decides what new info to write to cell state
Output Gate	σ(Wo·[hₜ₋₁, xₜ] + bo)	Decides what part of cell state to output as hₜ
Cell State Update	Cₜ = f⊙Cₜ₋₁ + i⊙tanh(Wc·[hₜ₋₁,xₜ]+bc)	Gated update of long-term memory

Gates use sigmoid (σ) — output between 0 and 1 — acting as a soft switch. LSTM can maintain context over hundreds of time steps.

GRU (Gated Recurrent Unit)

Simplified LSTM with only 2 gates (Reset + Update):

• Fewer parameters → trains faster; often comparable performance to LSTM
• GRU(units) in Keras

Bidirectional LSTM

Processes sequence both left-to-right AND right-to-left; concatenates both hidden states:

• Forward LSTM: sees past context
• Backward LSTM: sees future context
• Together: each token's representation is informed by its full surrounding context
• Especially powerful for classification tasks (sentiment, NER, sarcasm)
• Bidirectional(LSTM(units))

Sequence Padding

RNNs require fixed-length input sequences:

• pad_sequences(sequences, maxlen=500, padding='pre') — pad at the start (pre-padding is standard for LSTM)
• Shorter sequences get zero-padding; longer sequences are truncated
• maxlen is a hyperparameter: too short = information lost; too long = slower training, more padding noise

Embedding Layer

Learns dense word representations as part of the network (unlike pre-computed Word2Vec):

• Embedding(vocab_size, embedding_dim, input_length=maxlen)
• Converts integer token IDs to dense vectors; trained end-to-end with the rest of the model
• More flexible than static embeddings — learned for the specific task

Pre-trained Embeddings (GloVe)

Global Vectors for Word Representation — trained on global word-word co-occurrence statistics from large corpora:

• Captures semantic relationships as vector offsets
• embedding_dim = 100 or 300 common; load from pre-trained .txt file
• Build weight matrix from GloVe vectors → initialise Embedding layer with weights=[embedding_matrix], trainable=False
• Especially useful when training data is limited

Attention Mechanism

Core innovation that led to the Transformer architecture:

• Instead of compressing the entire input into a single fixed-length context vector, attention allows the model to "look back" at all encoder hidden states when generating each output token
• Attention Score = how relevant each encoder hidden state is to the current decoder position
• Attention Weights = softmax over scores → probability distribution over input positions
• Context Vector = weighted sum of encoder hidden states using attention weights
• Used in Neural Machine Translation (NMT): NMT with Attention.ipynb

Neural Machine Translation (NMT) with Attention

• Encoder — LSTM processes source sentence; outputs all hidden states + final context
• Decoder — LSTM generates target sentence token by token; attention weights computed at each step to focus on relevant source tokens
• Overcomes fixed-length bottleneck of seq2seq without attention

Advanced Language Models (Overview)

• ELMo (Embeddings from Language Models) — context-aware embeddings from bidirectional LSTMs; same word gets different embeddings depending on context
• BERT (Bidirectional Encoder Representations from Transformers) — transformer-based; pre-trained with Masked Language Modelling + Next Sentence Prediction; fine-tuned for downstream tasks; context flows in both directions simultaneously
• GPT — autoregressive transformer; pre-trained on next-word prediction; strong text generation
• Transformer Architecture — multi-head self-attention; no recurrence → parallelisable; position encodings replace positional information from sequential processing

---

🏆 Project — Sequential NLP: Sentiment & Sarcasm Classification (NLP-2)

Submitted: 20 February 2022

---

Part 1 — IMDB Movie Review Sentiment Analysis (LSTM)

Domain: Digital Content & Entertainment

Business Problem: Classify 50,000 IMDB movie reviews as positive or negative sentiment.

Dataset:

• 50,000 reviews, pre-encoded as integer sequences (word index = frequency rank)
• Vocabulary size: 10,000 most frequent words; index 0 = unknown word (OOV)
• maxlen = 500 tokens; pre-padded sequences

Model Architecture:

Embedding(10000, embedding_dim, input_length=500)
  → LSTM(units)
  → Dense(1, sigmoid)

Loss: Binary Crossentropy | Optimiser: Adam

Results:

Config	Training Acc	Validation Acc	Test Acc
Baseline LSTM	~98.4%	~86.9%	~85.2%
LSTM + longer maxlen	Minimal improvement	Minimal improvement	—
LSTM + more units	Minimal improvement	—	—
LSTM + stop word removal	No improvement	—	—

Findings:

• Baseline LSTM achieved good test accuracy (~85%)
• Model slightly overfitted (train 98% vs val 87%) — all 3 tuning attempts had minimal impact
• Main route to further improvement: increase training data size

---

Part 2 — News Headline Sarcasm Detection (Bidirectional LSTM + GloVe)

Domain: Social Media Analytics

Business Problem: Classify news headlines as sarcastic (The Onion) or genuine (HuffPost).

Dataset:

• 28,619 headlines (theonion.com + huffingtonpost.com)
• Target: is_sarcastic (binary); 12 duplicate headlines removed
• Feature: headline text only (article_link dropped)

Preprocessing:

1. clean_headline — tokenisation + lemmatisation + stemming applied
2. WordCloud generated for visual EDA
3. Sequences converted to integer indices (word-to-index mapping)
4. Vocabulary size computed from corpus
5. Padding applied to uniform sequence length

Embedding:

• GloVe pre-trained embeddings loaded → weight matrix built for vocabulary
• Embedding layer initialised with GloVe weights (trainable=False)

Model Architecture:

Embedding(vocab_size, glove_dim, weights=[glove_matrix], trainable=False)
  → Bidirectional(LSTM(units))
  → Dense(1, sigmoid)

Loss: Binary Crossentropy | Optimiser: Adam

Results:

• Limited epochs run (technical constraint)
• Bidirectional LSTM + GloVe intended to leverage both left and right context + pre-trained semantic knowledge
• Accuracy improvable with 10–15 epochs and larger LSTM units

---

🔑 Key Learnings

1. Sequential models capture order; BoW does not — "not good" and "good not" are identical to BoW but opposite in meaning; LSTM preserves this distinction
2. LSTM gates = selective memory — forget gate prevents irrelevant old information from persisting; input gate controls what new information matters; output gate controls what to expose
3. Pre-padding is standard for LSTM — padding at the start means the actual text (more informative) is closest to the LSTM output
4. Bidirectional LSTM for classification — full context (past + future) gives richer representations for tasks like sentiment and sarcasm where the whole sentence matters
5. GloVe embeddings help with small datasets — initialising with pre-trained weights gives the model semantic head start; trainable=False prevents forgetting during fine-tuning
6. Overfitting in LSTMs — adding Dropout after LSTM layers or using recurrent dropout is more effective than just tuning review length or units
7. Attention > fixed context vector — seq2seq without attention compresses everything into one vector; attention lets the decoder look at the full input, dramatically improving translation quality
8. BERT is bidirectional at training time — unlike stacked Bidirectional LSTMs, BERT's transformer attention sees all positions simultaneously, not just left or right

---

🛠️ Tools & Libraries

import re
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from wordcloud import WordCloud
import nltk
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer

import tensorflow as tf
from tensorflow import keras
from keras.models import Sequential, Model
from keras.layers import (Embedding, LSTM, GRU, Dense, Dropout,
                           Bidirectional, Attention)
from keras.optimizers import Adam
from keras.preprocessing.sequence import pad_sequences
from keras.preprocessing.text import Tokenizer

from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix