transformer-lens-interpretability

To inspect transformer activations for mechanistic interpretability, load a pretrained GPT-style model using HookedTransformer and run a forward pass with run_with_cache to collect an ActivationCache of residual streams and attention heads.

Activation patching is a mechanistic interpretability technique for isolating circuit-level behavior in transformer models. It works by intercepting HookPoints during a forward pass to overwrite intermediate activations and measure the resulting causal impact on outputs.

Yes, TransformerLens supports indirect object identification (IOI) workflows and circuit discovery. Researchers use its HookedTransformer API and run_with_hooks to trace attention head behaviors and logit attribution across GPT-family models.

Yes, TransformerLens supports logit attribution and induction head detection by caching intermediate activations and attention patterns. Researchers analyze the ActivationCache to map how specific components contribute to token predictions across layers.

The best way to analyze circuit-level dynamics in GPT-style transformers is using TransformerLens to run_with_hooks and apply activation patching. This isolates specific model components and traces their causal influence through the residual stream.

Steering experiments across large language models require loading a pretrained GPT-style transformer into the HookedTransformer architecture. You then use run_with_cache to capture residual stream activations before applying directional intervention techniques.