transformer-lens-interpretability

Inspect transformer activations, attention, and circuits with TransformerLens.

Updated Mar 16, 2026

Search Tags:#mechanistic-interpretability #causal-tracing #transformer-lens #hookpoints #logit-attribution #induction-heads #activation-caching

Ready to run

npx skills add https://github.com/arsity/scholar-tools --skill transformer-lens-interpretability-arsity

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: transformer-lens-interpretability
Source: https://github.com/arsity/scholar-tools/tree/main/vendor/ai-research-skills/04-mechanistic-interpretability/transformer-lens
Command: npx skills add https://github.com/arsity/scholar-tools --skill transformer-lens-interpretability-arsity

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Provides a principled framework for mechanistic interpretability research on GPT-style transformers, enabling researchers to inspect activations, attention patterns, and circuit-level dynamics using TransformerLens.

Core Features & Use Cases

Activation caching and HookPoint-based inspection of residual streams, attention heads, and MLP outputs.
Activation patching, IOI circuit discovery, and indirect object identification workflows.
Logit attribution, induction head detection, and steering experiments across large language models.

Quick Start

Load a pretrained model with HookedTransformer, run a forward pass to collect activations with run_with_cache, and begin exploring layer-wise patterns using the ActivationCache.

Frequently Asked Questions about transformer-lens-interpretability

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I inspect transformer activations and attention patterns for mechanistic interpretability?▼

To inspect transformer activations for mechanistic interpretability, load a pretrained GPT-style model using HookedTransformer and run a forward pass with run_with_cache to collect an ActivationCache of residual streams and attention heads.

What is activation patching and how does it work with HookPoints?▼

Activation patching is a mechanistic interpretability technique for isolating circuit-level behavior in transformer models. It works by intercepting HookPoints during a forward pass to overwrite intermediate activations and measure the resulting causal impact on outputs.

Can I use TransformerLens for indirect object identification and IOI circuit discovery?▼

Yes, TransformerLens supports indirect object identification (IOI) workflows and circuit discovery. Researchers use its HookedTransformer API and run_with_hooks to trace attention head behaviors and logit attribution across GPT-family models.

Does TransformerLens support logit attribution and induction head detection?▼

Yes, TransformerLens supports logit attribution and induction head detection by caching intermediate activations and attention patterns. Researchers analyze the ActivationCache to map how specific components contribute to token predictions across layers.

What is the best way to analyze circuit-level dynamics in GPT-style transformers?▼

The best way to analyze circuit-level dynamics in GPT-style transformers is using TransformerLens to run_with_hooks and apply activation patching. This isolates specific model components and traces their causal influence through the residual stream.

Do I need a specific environment setup to run steering experiments across large language models?▼

Steering experiments across large language models require loading a pretrained GPT-style transformer into the HookedTransformer architecture. You then use run_with_cache to capture residual stream activations before applying directional intervention techniques.