transformer-lens-interpretability

You can perform activation patching by caching activations from a clean prompt and intervening at specific HookPoints in TransformerLens. This identifies which layer and position combinations causally drive the output difference for corrupted prompts.

Mechanistic interpretability reverse-engineers transformer computations by inspecting cached activations. Activation caching captures residual streams, attention patterns, and per-head representations like resid_post and pattern tensors across layers for deterministic circuit analysis.

You analyze attention patterns by accessing the pattern activation cache keys via TransformerLens HookPoints. This allows you to inspect per-head attention weights across positions to detect induction heads and map circuit behaviors in GPT-style models.

Yes, mechanistic interpretability experiments with TransformerLens require PyTorch version 2.0.0 or higher and TransformerLens version 2.0.0 or higher. These dependencies provide the tensor computation and HookPoints infrastructure for activation patching workflows.

Yes, TransformerLens supports interpretable circuit analysis workflows including indirect object identification and logit attribution. You can trace how specific attention heads and layers contribute to final token predictions by inspecting cached residual streams.

Activation patching via HookPoints requires deterministic research pipelines and careful alignment of clean and corrupted prompts. Complex circuits may involve interactions across many layers and positions, making causal interpretation challenging without well-controlled experimental prompt pairs.