Interpretability in transformers

Beginner

When a transformer gives an answer, interpretability asks questions such as:

• Which input tokens mattered? For example, did the model rely on the right phrase in the prompt?
• Which internal components mattered? Did a particular attention head, MLP neuron group, or layer contribute strongly?
• What information is represented? Can hidden states encode sentiment, syntax, named entities, or factual recall cues?
• Did the model use a shortcut? Is it relying on formatting, position, or spurious patterns instead of the intended evidence?

Common families of interpretability tools

Prompt tokens
   -> embeddings
   -> repeated transformer layers
   -> hidden states, heads, MLP activations
   -> logits
   -> output token

Interpretability asks what can be learned at each arrow or internal state.

tools = ["attention visualization", "attribution", "probing", "intervention"]
print(tools)

Simple example: suppose a transformer classifier labels support emails as urgent. Interpretability can help check whether it is responding to the actual complaint text or merely to superficial signals such as all-caps words, signatures, or a ticket template.

Intermediate

At intermediate depth, it helps to separate descriptive methods from causal methods.

Descriptive vs causal evidence

• Descriptive methods summarize what the model appears to focus on or encode. Examples: attention maps, logit lens, representation similarity, and probes.
• Causal methods test what changes model behavior. Examples: head ablation, neuron ablation, activation patching, and counterfactual replacement.
• Descriptive methods are often easier and faster.
• Causal methods are usually stronger when you need engineering confidence.

Important transformer objects to inspect

• Attention heads: sometimes specialize in copying, induction-like patterns, delimiter tracking, or syntactic behavior.
• MLP activations: often contain feature detectors or memory-like behavior, but can be highly polysemantic.
• Residual stream states: the running representation passed between layers; many tools inspect how information accumulates here.
• Logits and logit contributions: useful when asking which internal component most changed the next-token distribution.

Widely used techniques

Example: a simple probe

from sklearn.linear_model import LogisticRegression

# X: hidden states from one transformer layer
# y: labels such as whether the token is inside a named entity
probe = LogisticRegression(max_iter=2000)
probe.fit(X_train, y_train)

score = probe.score(X_valid, y_valid)
print(f"probe accuracy: {score:.3f}")

A good probe result means the representation contains usable information about the label. It does not by itself prove that the model relies on that information when producing its final answer.

Example: activation patching idea

clean_run = model("The capital of France is Paris")
corrupted_run = model("The capital of France is Rome")

patched = patch_activation(
    source=clean_run,
    target=corrupted_run,
    layer=10,
    position=-1,
)

print(read_top_token(patched))

If patching one activation from the clean run into the corrupted run restores the correct prediction, that is strong evidence that the patched location carried important information for the answer.

Advanced

Advanced interpretability aims for mechanistic understanding: not just observing correlations, but describing how parts of the transformer cooperate to implement a computation. In the strongest version, you can name a circuit, intervene on it, and predict the effect of the intervention.

Mechanistic interpretability

• Circuits: small sets of components whose interactions implement a behavior, such as copying, induction, or bracket matching.
• Feature directions: internal directions in activation space that correspond to concepts or reusable pieces of computation.
• Superposition: models may pack multiple features into the same neurons or directions, which makes simple neuron-level interpretation incomplete.
• Polysemanticity: one neuron or direction may mix unrelated features, which complicates naive “one neuron = one concept” stories.

Input prompt
   -> early layers build local and lexical features
   -> middle layers combine and route information
   -> later layers sharpen task-specific predictions
   -> logits

Mechanistic interpretability asks which features and components form the path
from evidence in the prompt to the final logit change.

What strong evidence looks like

• You can identify a component or feature hypothesis.
• You can predict what happens when it is ablated, patched, or amplified.
• The intervention produces the expected behavioral change across multiple examples, not just one cherry-picked case.
• The explanation remains stable enough to support debugging or model comparison.

Typical failure modes in interpretability work

• Storytelling from pretty plots: attention heatmaps and neuron dashboards can be suggestive but not decisive.
• Probe overclaiming: a probe can succeed because the information is somewhere in the representation, even if no downstream component uses it.
• Single-example overfitting: a convincing explanation on one prompt may fail on a broader sample.
• Redundancy blindness: ablation may show little effect because the model has backups, not because a component is unimportant.
• Distribution mismatch: methods that work on toy models or synthetic tasks may weaken on messy real prompts.

Practical uses for engineers and researchers

• Find spurious shortcut features after fine-tuning.
• Compare how internal behavior changes between base and instruction-tuned models.
• Diagnose why retrieval or tools are being ignored in a larger pipeline.
• Check whether a model is storing task-relevant information earlier than expected.
• Localize components worth pruning, monitoring, or studying further.

A compact workflow

questions = [
			    "What feature or behavior am I trying to explain?",
			    "What descriptive signal suggests a hypothesis?",
			    "What causal intervention can test it?",
			    "Does the result generalize beyond one prompt?"
			]
print(questions)