Word embeddings

Core concept

One-hot vs dense embeddings

One-hot vectors are sparse and encode no similarity: every word is the same distance from every other. Dense embeddings compress words into low-dimensional vectors so words used in similar contexts land near each other.

One-hot  "cat" -> [0, 0, 0, 1, 0, 0, ...]   sparse, orthogonal, no similarity
Embedding "cat" -> [0.21, -0.44, 0.83, ...]  dense, cosine similarity is meaningful

Distributional hypothesis: words with similar contexts get similar vectors.

• Dimensionality: typically 50–300 dimensions vs vocabulary-size one-hot vectors.
• Pre-trained: embeddings trained on large corpora can be reused as features (transfer learning before transformers).
• Static: one vector per word type regardless of sentence context — the core limitation compared to contextual models.

Word2Vec

Two training objectives

Word2Vec is a shallow two-layer neural network trained on raw text. The weights of the embedding layer become the learned vectors. Two variants exist and interviewers frequently ask you to compare them.

CBOW:     [the, __, sat]  -> "cat"   (context -> center)
Skip-gram: "cat"          -> [the, sat]  (center -> context)

Negative sampling (key training trick)

Training a softmax over the entire vocabulary is expensive. Negative sampling turns the problem into binary classification: for each real context pair, sample k random "negative" words and train the model to score real pairs higher than random ones. Typical k = 5–20 for small datasets, 2–5 for large.

# Conceptual objective (not exact code)
# Maximize: log sigmoid(v_context · v_center)
# +  sum of k negatives: log sigmoid(-v_neg · v_center)

# Hierarchical softmax is an alternative: uses a binary tree
# over vocabulary — O(log V) instead of O(V) per update.

Hyperparameters that matter

• Window size: small windows (2–5) capture syntactic relationships; large windows (10+) capture more topical/semantic similarity.
• Embedding dimension: 100–300 is common; larger is not always better.
• Subsampling frequent words: words like "the" are randomly dropped during training to balance signal.

GloVe

Global co-occurrence factorization

GloVe (Global Vectors) builds an explicit word–word co-occurrence matrix from the whole corpus, then factorizes it so that the dot product of two word vectors approximates the log of their co-occurrence probability.

Step 1: Build co-occurrence matrix X  (X_ij = how often word j appears near word i)
Step 2: Minimize:  sum_ij f(X_ij) (v_i · v_j + b_i + b_j - log X_ij)^2

f(X_ij) = weighting function that down-weights very frequent pairs.

• Difference from Word2Vec: GloVe uses global corpus statistics directly; Word2Vec learns from local context windows iteratively.
• In practice: both produce similar quality vectors; GloVe is often faster to train on a single machine once the matrix is built.
• Still static: one vector per word, same polysemy limitation as Word2Vec.

fastText

Subword character n-grams

fastText (Facebook, 2016) represents each word as the sum of its character n-gram vectors. For example, "playing" is decomposed into <pla, lay, ayi, yin, ing, ng> plus the full word token.

• OOV handling: unknown words still get a vector by composing their n-grams — a major advantage over Word2Vec and GloVe.
• Morphology: "run", "runs", "running" share n-grams so their vectors are naturally close.
• Misspellings: "colour" and "color" share most n-grams, reducing sensitivity to spelling variation.
• Tradeoff: larger model size and longer training vs plain Word2Vec.

How fastText Handles Out-of-Vocabulary (OOV) Words

Word2Vec and GloVe assign one vector per word in a fixed vocabulary. If a word wasn't seen during training, it gets no vector at all — it is OOV. fastText solves this by never storing just whole-word vectors; instead, every word is represented as the sum of its character n-gram vectors.

Step-by-step: how it works

1. Decompose into character n-grams — the word is wrapped with boundary markers (< and >) and split into all n-grams of length 3 to 6 (by default).
2. Look up each n-gram's vector — every n-gram has its own learned embedding in a shared table.
3. Sum them up — the final word vector is the sum (or average) of all its n-gram vectors plus the whole-word vector if available.

Known word: "playing"
  Boundary:  <playing>
  N-grams:   <pl, pla, lay, ayi, yin, ing, ng>, play, layi, ayin, ying, ...
  Vector:    sum of all n-gram vectors + whole-word vector

OOV word: "playying" (misspelling, never seen in training)
  Boundary:  <playying>
  N-grams:   <pl, pla, lay, ayy, yyi, yin, ing, ng>, play, layy, ...
  Vector:    sum of all n-gram vectors (no whole-word vector exists)

Most n-grams overlap with "playing" → the OOV vector is close to the known word!

Why this works so well

• Morphological similarity — "run", "runs", "running", "runner" share n-grams like "run", "<ru", "un>", so their vectors cluster together automatically
• Misspelling robustness — "recieve" shares most n-grams with "receive", so the vectors are close despite the typo
• New/rare words — technical terms, slang, or brand names get reasonable vectors from their character structure alone
• Multilingual strength — languages with rich morphology (Arabic, Turkish, Finnish) benefit hugely since word forms share roots

Comparison: OOV handling across methods

Similarity and analogies

Cosine similarity

Vector direction is more informative than magnitude, so embeddings are compared with cosine similarity rather than Euclidean distance.

def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    norm_a = sum(x * x for x in a) ** 0.5
    norm_b = sum(x * x for x in b) ** 0.5
    return dot / (norm_a * norm_b) if norm_a and norm_b else 0.0

king  = [0.20,  0.90, -0.10]
queen = [0.19,  0.88, -0.08]
apple = [-0.50, 0.11,  0.72]

print(cosine_similarity(king, queen))   # high (~0.999)
print(cosine_similarity(king, apple))   # low

Vector arithmetic and analogies

Well-trained embedding spaces exhibit linear regularities. The canonical example:

king - man + woman ≈ queen

Paris - France + Italy ≈ Rome   (capital-country relationship)
bigger - big + small  ≈ smaller (comparative morphology)

Key limitations — interview must-knows

Quick comparison