BERT and Transformer Pretraining

Beginner

BERT is trained by hiding some tokens and asking the model to recover them from both left and right context. That bidirectional setup helped transformers learn much richer representations for sentence classification, token labeling, extractive question answering, and semantic matching than earlier feature-based pipelines.

Why BERT mattered

Pretraining first, supervision second: the model learns generic syntax and semantics from unlabeled text, then adapts with relatively little labeled data.
Bidirectional context: each masked token can use both the left and right side of the sentence, which is especially useful for understanding tasks.
One backbone, many tasks: the same pretrained encoder can be fine-tuned for sentiment analysis, natural language inference, named entity recognition, and extractive QA.
Encoder focus: BERT is built to understand text representations, not to generate long free-form answers token by token.

What the model actually sees

BERT input is more structured than plain text. A sequence is converted into subword tokens, wrapped with special markers, and represented by the sum of token, segment, and position embeddings.

Sentence A            Sentence B
    |                     |
WordPiece tokens     WordPiece tokens
    |                     |
[CLS] ... tokens ... [SEP] ... tokens ... [SEP]
   + token embeddings + segment embeddings + position embeddings

[CLS] is a special classification token whose final representation is often used for sentence-level prediction.
[SEP] separates segments such as sentence pairs.
Segment embeddings tell the model whether a token belongs to sentence A or sentence B.
WordPiece tokenization breaks rare words into reusable subword units so vocabulary size stays manageable.

Core pretraining idea

The central BERT objective is masked language modeling. Some tokens are selected for prediction, and the model learns to infer them from context rather than from hand-built linguistic rules.

sentence = "The capital of France is Paris."
tokens = ["[CLS]", "the", "capital", "of", "france", "is", "[MASK]", ".", "[SEP]"]
target = "paris"

print(tokens)
print("predict:", target)

Real-world example: if you fine-tune BERT for customer-support intent classification, you usually need far less labeled data than if you trained an encoder from scratch because pretraining already gave the model a useful language representation space.

Pretraining and fine-tuning workflow

Large unlabeled text
        |
        v
Masked-language-model pretraining
        |
        v
Pretrained BERT checkpoint
        |
        +--> Add classifier head for sentiment / NLI
        +--> Add token head for tagging / NER
        +--> Add span head for extractive QA

Simple mental model: pretraining builds general-purpose language features; fine-tuning reshapes those features for one concrete downstream objective.

Advanced

Pretraining objectives and training recipes determine what an encoder learns. BERT combined masked language modeling with next sentence prediction, while later work such as RoBERTa showed that longer training, more data, bigger batches, and revised masking strategy could outperform many supposedly architectural improvements.

BERT pretraining objectives

Masked language modeling (MLM): select a subset of tokens and predict the original identity of those tokens from context.
Original masking recipe: BERT selects 15% of tokens for prediction; of those selected tokens, 80% are replaced with [MASK], 10% with a random token, and 10% are left unchanged.
Next sentence prediction (NSP): a sentence-pair objective intended to help the model reason about inter-sentence relationships.
Transfer effect: once pretrained, the encoder can usually be adapted by adding only a small task head rather than designing a custom network per task.

Why the 80/10/10 trick exists: always replacing with [MASK] would make pretraining too different from downstream inference, where [MASK] never appears.

Recipe details that changed after BERT

Model	What changed	Why it mattered
BERT	MLM + NSP with static masking and standard pretraining schedule.	Established the encoder pretrain-then-fine-tune paradigm.
RoBERTa	Removed NSP, trained longer, used larger batches, more data, and dynamic masking.	Showed BERT had been undertrained and that recipe quality mattered a lot.
ALBERT	Parameter sharing and factorized embeddings; replaced NSP with sentence order prediction.	Reduced parameter count while preserving strong transfer performance.
DistilBERT	Compressed a BERT-style model with distillation after pretraining.	Made encoder transfer cheaper at inference time with moderate quality loss.

Batch size in pretraining

Batch size was one of the most important recipe variables across BERT-family models. Increasing it was a key factor in RoBERTa's gains over the original BERT.

Model	Batch size (sequences)	Notes
BERT-Base	256	Original recipe; later shown to be suboptimal
RoBERTa	8,192	32× larger; combined with more data and longer training for significant gains
GPT-3	~3.2M tokens per step	Gradually increased during training using a warmup schedule

Larger batches improved pretraining perplexity by giving more stable gradient estimates per update, allowing higher learning rates and faster convergence.
RoBERTa's lesson: BERT's original batch size of 256 was too small — scaling it to 8,192 (with matched learning rate) was one of the easiest wins in the improved recipe.
Batch size warmup: GPT-3 started with smaller batches early in training (when the loss landscape is steep) and gradually increased batch size — this improved stability and final quality.

Key takeaway: in pretraining, batch size is not just a hardware convenience — it is a first-class hyperparameter that directly affects model quality. Larger batches with proper learning rate scaling consistently outperformed smaller ones across the BERT family.

Static versus dynamic masking

In static masking, each training example tends to reuse the same masked positions across epochs. In dynamic masking, the masked tokens can change each time the example is seen. Dynamic masking exposes the model to more prediction targets and generally uses the corpus more efficiently.

Same sentence across epochs

Static masking:
Epoch 1: The capital of France is [MASK].
Epoch 2: The capital of France is [MASK].

Dynamic masking:
Epoch 1: The capital of [MASK] is Paris.
Epoch 2: The [MASK] of France is Paris.

What gets transferred downstream

BERT-style pretraining gives you contextual token representations, sentence-level pooled representations, and layer-wise features that can be reused in different ways depending on the task.

Sequence classification: use the final [CLS] representation or a pooled variant for labels such as sentiment or topic.
Token classification: read the per-token hidden states for tasks like named entity recognition or part-of-speech tagging.
Sentence-pair reasoning: package two texts with separator tokens for entailment, duplicate detection, or ranking.
Extractive QA: predict answer start and end positions over the input tokens rather than generating an answer freely.

Tokenized corpus -> MLM / sentence-level objective -> pretrained encoder
       |
       +--> frozen features
       +--> full fine-tuning
       +--> task-specific head

Minimal fine-tuning pattern

from transformers import AutoTokenizer, AutoModelForSequenceClassification

checkpoint = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)

batch = tokenizer(
    ["the service was excellent", "the refund process was confusing"],
    padding=True,
    truncation=True,
    return_tensors="pt",
)

outputs = model(**batch)
print(outputs.logits.shape)

Limits and common misunderstandings

BERT is not a chat model: encoder pretraining gives strong representations, but it does not make the model naturally autoregressive or instruction-following.
Masking creates a train-test mismatch: downstream inference rarely contains explicit mask tokens.
Corpus quality matters: more text helps only when the text is diverse, clean enough, and reasonably matched to the target domain.
Sequence length constraints matter: classic BERT uses a fixed context window and may struggle on long documents without careful chunking or specialized variants.
Recipe improvements are not always architectural breakthroughs: RoBERTa is the standard cautionary example that training longer and better can beat supposedly newer ideas.

Keep this module separate from general transformer mechanics and decoder-only LLM behavior. The core topic here is encoder pretraining, objective design, and transfer learning around the BERT family.

Understanding BERT matters because it explains the modern transfer-learning playbook: start with a broad self-supervised objective, learn reusable representations at scale, and adapt them efficiently to downstream tasks.

Where BERT Is Used in Production

Search ranking — Google uses BERT to understand query intent; many companies use BERT-based cross-encoders to re-rank search results.
Sentiment & intent classification — customer support systems classify tickets, detect urgency, and route conversations using fine-tuned BERT.
Named Entity Recognition (NER) — extracting names, dates, amounts, and entities from documents (KYC, legal, medical).
Spam & toxicity detection — email providers and social platforms use BERT classifiers for content moderation.
Semantic similarity — duplicate detection, FAQ matching, and paraphrase identification using Sentence-BERT (SBERT) embeddings.
Extractive QA — given a passage, BERT predicts the exact span that answers a question (used in internal knowledge bases).

Why BERT and not an LLM? BERT models are small (110M–340M params), fast, cheap to run, and excel at classification and ranking. When you need a label or a score — not free-form text — BERT is often the better production choice.

BERT's Role in RAG

In Retrieval-Augmented Generation, BERT-family models power two critical stages:

1. Embedding & retrieval (bi-encoder)

Models like Sentence-BERT or E5 encode documents and queries into dense vectors. At query time, the nearest vectors are retrieved from a vector database.

Documents → BERT encoder → dense vectors → stored in vector DB
Query     → BERT encoder → query vector  → ANN search → top-k docs

2. Re-ranking (cross-encoder)

A BERT cross-encoder takes (query, document) pairs and scores relevance jointly. This is slower but more accurate than bi-encoder retrieval alone, so it is used to re-rank the top-k results.

RAG pipeline:
  Query → bi-encoder retrieval (fast, top-100)
        → cross-encoder re-rank (accurate, top-5)
        → feed top docs into LLM as context
        → LLM generates answer

Why BERT fits RAG

Encoding speed — BERT embeddings are fast enough for real-time retrieval at scale.
Separation of concerns — BERT handles understanding and ranking; the LLM handles generation. Each does what it's best at.
Cost — running a BERT encoder is orders of magnitude cheaper than passing every document through an LLM.

Interview point: in a RAG system, BERT is the retriever, not the generator. It finds the right context so the LLM can produce a grounded answer. This is one of the most common production uses of BERT-family models today.

Practice

Tokenize several sentence pairs and inspect where [CLS], [SEP], and segment IDs appear.
Run masked-token prediction examples and check whether the model uses both left and right context sensibly.
Fine-tune a small BERT checkpoint on one classification task and compare it with a linear baseline.
Compare frozen-encoder features against end-to-end fine-tuning on the same dataset.

Build

Create a sentence-classification app backed by a pretrained BERT-family encoder.
Build an experiment sheet comparing BERT, RoBERTa, and a non-pretrained baseline on the same task.
Write a short technical note explaining why encoder pretraining changed NLP transfer learning.
Package a reproducible fine-tuning workflow with saved tokenizer, checkpoint, metrics, and inference script.