transformer-lens-interpretability

Inspect transformer internals by utilizing HookPoints to access and cache attention patterns during model inference. This mechanism enables researchers to directly observe and analyze how transformer models process and route input data.

Activation patching is a mechanistic analysis technique for causal tracing in transformer models. It works by replacing specific cached activations during inference to isolate and identify which internal components drive specific model behaviors.

Yes, you need PyTorch installed. Performing transformer interpretability experiments requires torch>=2.0.0 and transformer-lens>=2.0.0 as foundational dependencies to manipulate model internals and execute activation caching operations.

Yes, you can reverse-engineer transformer algorithms. The mechanistic interpretability tools facilitate algorithm reverse-engineering by allowing researchers to inspect, manipulate, and trace causal pathways within the transformer model internals.

The best way to run causal tracing is to execute an interpretability experiment on text data using default settings. This setup leverages HookPoints to cache and patch activations, systematically identifying the causal mechanisms behind model outputs.

Activation patching may not be suitable when you lack the required transformer-lens and torch dependencies, or when your model architecture does not support HookPoints. It is strictly designed for mechanistic analysis of transformer models.