01 · Text Data & Tokenization 🔤

4 min read

A neural network only speaks numbers. Before it can read a sentence, you have to translate that sentence into a tensor — and how you do that shapes everything downstream.

1. The 'Why'

Every model you've built consumes tensors of numbers, but text arrives as strings — sequences of characters with no numeric meaning a network can use. Bridging that gap is the unglamorous but foundational first step of all NLP, and getting it wrong makes everything after it impossible. You can't matmul the word "cat." So the entire pipeline begins with a translation problem: how do we turn "the cat sat" into something like [4, 17, 92] that a model can process, in a way that's consistent, reversible, and handles the messiness of real language — punctuation, capitalization, unknown words, and sentences of wildly different lengths?

The answer is a three-stage pipeline that's worth understanding explicitly even though libraries automate it. First, tokenization splits raw text into units (tokens) — words, sub-words, or characters. Second, a vocabulary maps each unique token to an integer ID, giving you numericalization. Third, because sentences vary in length but tensors must be rectangular, you pad shorter sequences so a batch stacks into a single tensor (exactly the collate_fn scenario from Module 04). These integer IDs aren't the final representation — they're indices that the embedding layer (sub-module 02) will turn into meaningful vectors — but without this clean text-to-tensor plumbing, nothing else in NLP can run.

2. Core Concepts

Tokenization. Splitting text into tokens. Word-level is intuitive but has huge vocabularies and chokes on unseen words. Sub-word (BPE, WordPiece) — used by modern models — splits rare words into pieces, balancing vocabulary size and coverage. Character-level is tiny but makes sequences long.

Vocabulary & numericalization. A dictionary mapping each token to a unique integer ID. Applying it converts a token list into an ID list. You also reserve special tokens: <pad> for padding, <unk> for unknown words, often <bos>/<eos> for sequence boundaries.

Padding. To batch sequences of different lengths, pad the short ones (usually with the <pad> ID) up to a common length. torch.nn.utils.rnn.pad_sequence does this.

Attention/padding masks. Because padding is filler, you often pass a mask telling the model which positions are real, so it ignores the pad tokens. This becomes essential for LSTMs and Transformers.

3. Code in Action

A tiny tokenizer + vocabulary

# 1. Tokenize (naive whitespace/lowercase tokenizer for illustration)
def tokenize(text):
    return text.lower().replace(".", " .").split()

corpus = ["The cat sat.", "The dog ran."]
tokenized = [tokenize(s) for s in corpus]
print(tokenized)   # [['the', 'cat', 'sat', '.'], ['the', 'dog', 'ran', '.']]

# 2. Build a vocabulary (reserve special tokens first)
specials = ["<pad>", "<unk>"]
vocab_tokens = specials + sorted({t for sent in tokenized for t in sent})
stoi = {tok: i for i, tok in enumerate(vocab_tokens)}   # string -> index
itos = {i: tok for tok, i in stoi.items()}              # index -> string
print(stoi)   # {'<pad>': 0, '<unk>': 1, '.': 2, 'cat': 3, ...}

Numericalizing text (with `<unk>` fallback)

def numericalize(tokens, stoi):
    # unknown words map to <unk> so inference never crashes on new vocabulary
    return [stoi.get(tok, stoi["<unk>"]) for tok in tokens]

ids = [numericalize(sent, stoi) for sent in tokenized]
print(ids)     # e.g. [[7, 3, 6, 2], [7, 4, 5, 2]] -> integer sequences

Padding a batch into a rectangular tensor

import torch
from torch.nn.utils.rnn import pad_sequence

seqs = [torch.tensor(s) for s in ids]                    # variable-length tensors
padded = pad_sequence(seqs, batch_first=True, padding_value=stoi["<pad>"])
print(padded.shape)   # torch.Size([2, 4]) -> (batch, max_len)

# A mask marks real tokens (True) vs padding (False) for the model to respect
mask = padded != stoi["<pad>"]
print(mask)

4. Common Pitfalls

Pitfall 1 — Building the vocabulary from the whole dataset (leakage). Fit stoi on the training split only. Including validation/test text lets information leak and inflates results — the same leakage warning from Module 04, applied to vocabulary.

Pitfall 2 — No <unk> handling at inference. Real inputs contain words your vocabulary never saw. If numericalize does a bare dictionary lookup, it crashes on the first unknown word. Always map out-of-vocabulary tokens to <unk>.

Pitfall 3 — Forgetting the padding mask. If you pad but don't tell the model which positions are padding, it treats <pad> as real content, corrupting the output (especially in pooling and attention). Carry a mask alongside the padded tensor.

5. Further Reading & Watch List

➡️ Next: 02 · Embeddings