What problem does it solve?
Enables researchers and engineers to access, analyze, and modify internal activations and module outputs of PyTorch models so interpretable causal experiments can be run on small local models or massive remote models without local GPU resources.
Core Features & Use Cases
- Activation analysis: Collect and inspect hidden states and attention patterns across layers for diagnostic and research purposes.
- Activation patching & interventions: Replace or modify layer activations during generation to test causal effects on model outputs.
- Remote execution via NDIF: Run identical interpretability code locally or remotely on large foundation models (70B+) by toggling remote=True and using NDIF sessions.
- Cross-prompt sharing & generation steering: Share activations between prompts, perform in-trace interventions during generation, and compute steering vectors for behavior modification.
- Gradient-based inspection: Retain gradients for saved activations to support attribution and influence analyses (not supported with vLLM or remote execution).
Quick Start
Use nnsight to run an activation-patching experiment by loading a PyTorch model with LanguageModel, collecting a target layer's activations with .save() in a trace, and then re-injecting those activations into a second trace (enable remote=True to run on NDIF).
