Classical NLP Techniques That Still Matter

What's Changed in the LLM Era?

LLMs handle most of the heavy lifting now, but foundational techniques still give you an edge — whether it's debugging a model, optimising performance, or implementing pragmatic solutions where full-scale LLMs are overkill.

Text Preprocessing Pipeline

Even in the LLM era, data quality matters. Garbage in, garbage out.

• Tokenization — breaking text into words, phrases, or subwords. SentencePiece and HuggingFace FastTokenizers are widely used
• Stopword Removal — removing common words like "is" and "the". Modern models often handle this implicitly, but it still helps in traditional pipelines
• Normalization — converting text to a standard format: lowercasing, stemming, lemmatization, handling special characters
• Text Cleaning — removing unwanted symbols, HTML tags, and noise
• Handling Emojis & Special Characters — some applications replace emojis with text descriptions to retain meaning

Raw text: "<p>I LOVE this product!!! 😊</p>"

Step 1 — Clean:       "I LOVE this product!!!"        (remove HTML)
Step 2 — Normalize:   "i love this product"            (lowercase, strip punctuation)
Step 3 — Tokenize:    ["i", "love", "this", "product"] (split into tokens)
Step 4 — Stopwords:   ["love", "product"]              (remove "i", "this")

What Are Subwords?

A subword is a unit smaller than a full word but larger than a single character. Instead of treating each word as one token or splitting into individual characters, subword tokenization finds a middle ground by breaking words into meaningful pieces.

Why Subwords?

Word-level tokenization has a fundamental problem: the vocabulary is fixed. Any word not in the vocabulary becomes an unknown token (<UNK>). Subword tokenization solves this by splitting rare or unseen words into known pieces.

Word-level tokenization (fixed vocabulary):
  "unhappiness"  →  <UNK>           (not in vocabulary!)

Subword tokenization:
  "unhappiness"  →  ["un", "happi", "ness"]   (all pieces are known)
  "playing"      →  ["play", "ing"]
  "ChatGPT"      →  ["Chat", "G", "PT"]

Character-level tokenization:
  "cat"          →  ["c", "a", "t"]            (too granular, loses meaning)

Common Subword Algorithms

• Byte-Pair Encoding (BPE) — starts with characters, then iteratively merges the most frequent pairs. Used by GPT models
• WordPiece — similar to BPE but merges based on likelihood, not just frequency. Used by BERT
• Unigram (SentencePiece) — starts with a large vocabulary and prunes it down. Used by T5 and multilingual models

Why This Matters

• Open vocabulary — any word can be represented, even misspellings and new words
• Morphology awareness — shared prefixes and suffixes ("un-", "-ing", "-tion") get their own tokens, so the model learns their meaning
• Compact vocabulary — typically 30k–50k tokens cover virtually all text, versus millions of unique words

Stemming vs Lemmatization

Both techniques reduce words to a base form to group related words together, but they work very differently.

Stemming

Stemming uses crude heuristic rules to chop off word endings. It is fast but often produces non-real words.

• How it works — applies suffix-stripping rules (e.g. remove "-ing", "-ed", "-ly")
• Popular algorithm — Porter Stemmer, Snowball Stemmer
• Pros — very fast, no dictionary needed
• Cons — can produce invalid words ("studies" → "studi", "running" → "run", but "university" → "univers")

Lemmatization

Lemmatization uses a dictionary and morphological analysis to return the actual base form (lemma) of a word. It is slower but always produces valid words.

• How it works — looks up the word in a vocabulary and applies morphological rules, often considering the part of speech
• Popular tools — WordNet Lemmatizer (NLTK), spaCy lemmatizer
• Pros — returns real dictionary words ("better" → "good", "was" → "be")
• Cons — slower, needs POS tagging for best results

Word          Stemming         Lemmatization
─────────     ───────────      ──────────────
"running"  →  "run"         →  "run"
"better"   →  "better"      →  "good"
"studies"  →  "studi"       →  "study"
"was"      →  "wa"          →  "be"
"flies"    →  "fli"         →  "fly"
"caring"   →  "car"         →  "care"

When to Use Which?

Bag of Words (BoW) and TF-IDF

Once text is preprocessed, it needs to be converted into a numerical format that ML models can process.

Bag of Words (BoW)

Represents text by counting word occurrences, ignoring grammar and order. Simple but effective for traditional NLP tasks like text classification.

TF-IDF

Improves upon BoW by assigning higher weights to rare but important words and lower weights to common words.

Related Concepts

• N-grams & Skip-grams — instead of single words, capture short sequences to provide more context
• Sparse Matrices — since most texts only contain a small subset of all possible words, sparse representations (like CSR format) save memory
• Feature Hashing — converts words into fixed-length numerical features, reducing memory for large datasets

Word Embeddings

BoW and TF-IDF treat words as independent entities, ignoring relationships between them. Word embeddings represent words in a continuous vector space, capturing their meanings and relationships.

Word2Vec (Google, 2013)

Learns word meanings based on context using two methods:

• Continuous Bag of Words (CBOW) — predicts a word from its surrounding context. Faster to train, works well with frequent words
• Skip-gram — predicts the surrounding context given a word. Works better for infrequent words

CBOW:      context words → [model] → target word
           "the ___ sat" → predicts "cat"

Skip-gram: target word → [model] → context words
           "cat" → predicts "the", "sat", "on", "mat"

GloVe (Stanford)

Instead of predicting words, GloVe learns from word co-occurrence matrices. It captures global statistics of word relationships, making it more stable in some cases.

FastText (Facebook)

Improves Word2Vec by using subword information (character n-grams), making it better at handling rare words or different word forms (e.g. "running" and "runner").

Recurrent Neural Networks (RNNs)

RNNs process text sequentially, maintaining memory of previous words. However, they suffer from the vanishing gradient problem — important information from earlier parts of a sequence gets lost as it travels through long chains of computations.

Input:  x₁ → x₂ → x₃ → ... → xₙ
         ↓     ↓     ↓            ↓
State:  h₁ → h₂ → h₃ → ... → hₙ  → output

Each hidden state depends on the previous one
→ sequential, cannot be parallelised
→ gradients shrink over long sequences (vanishing gradient)

LSTM (Long Short-Term Memory)

LSTMs solve the vanishing gradient problem using gates:

• Input gate — decides what new information to add to the cell state
• Forget gate — removes irrelevant information from the cell state
• Output gate — decides what information to output from the cell state

GRU (Gated Recurrent Unit)

GRUs are a simplified version of LSTMs, using fewer parameters (two gates instead of three) while maintaining similar performance. Faster to train.

CNNs for NLP

While CNNs were originally designed for image processing, they are surprisingly effective for NLP tasks like text classification, sentiment analysis, and NER. Instead of processing sequences word by word like RNNs, CNNs detect patterns using filters.

• 1D Convolutions — capture relationships between nearby words
• Character-level CNNs — work directly on characters, useful for noisy data (tweets, misspellings)
• CNN-RNN Hybrid Models — combine CNNs for feature extraction with RNNs for sequential dependencies

Input: "I love this movie"

Filters slide across the text:
  [I love]     → feature 1
  [love this]  → feature 2
  [this movie] → feature 3

Pool the strongest signals → classification

CNNs were fast and efficient, making them a popular choice for NLP before Transformers took over.

Sequence-to-Sequence (Seq2Seq) Models

The Seq2Seq architecture was a game-changer for translation, text summarisation, and chatbots. It introduced an encoder-decoder structure where the encoder processes input text into a hidden representation, and the decoder generates output text step by step.

Encoder                              Decoder
"I love NLP" → [h₁, h₂, h₃] → context vector → "J'adore le TAL" → <EOS>

Key Improvements

• Attention Mechanisms — helped models focus on relevant words in longer sequences instead of compressing everything into one vector
• Beam Search — allowed models to explore multiple output candidates, generating more fluent text

Limitations

• Still suffered from long-term dependency issues on very long sequences
• Sequential processing — difficult to parallelise, making training slow

These limitations directly motivated the development of the Transformer.

The Transformer: A New Era

In 2017, Transformers changed NLP forever. Introduced in the paper "Attention Is All You Need" (Vaswani et al.), Transformers solved everything Seq2Seq struggled with:

• Self-Attention Mechanism — lets models look at all words in a sentence simultaneously, rather than step by step
• Positional Encoding — preserves word order without requiring sequential processing
• Massive Parallelisation — made training incredibly fast and efficient on GPUs

Evolution of NLP Architectures:

RNNs (sequential)
  ↓ solved vanishing gradients
LSTMs / GRUs (gated sequential)
  ↓ added attention to encoder-decoder
Seq2Seq + Attention (still sequential encoder)
  ↓ replaced recurrence entirely with self-attention
Transformers (fully parallel)
  ↓ scaled up
BERT, GPT, T5, LLMs (modern era)

This led to models like BERT (encoder-only, understanding tasks), GPT (decoder-only, generation tasks), and T5 (encoder-decoder, both), which now power almost every modern NLP system.

When to Use What