What problem does it solve? Reverse-engineering the internal computations of large language models requires decomposing dense MLP activations into interpretable components. This Skill trains transcoders to decompose MLP sublayers into sparse, interpretable features and provides tools for circuit analysis, activation patching, and feature visualization on models like GPT-2 and Pythia. ## Core Features & Use Cases - Transcoder Training: Train sparse transcoders on any MLP sublayer using configurable hyperparameters (L1 coefficient, expansion factor, learning rate) with optional Weights & Biases logging. - Circuit Analysis: Compute feature-level attribution scores to identify which transcoder features drive specific model outputs, using replacement contexts to patch MLP layers during inference. - Feature Dashboards: Generate dashboards showing max-activating examples, top promoted/suppressed tokens, and activation distributions for individual features. - Use Case: Load pretrained GPT-2 transcoder weights from HuggingFace, run inference with a TranscoderReplacementContext, then use get_circuit_scores to trace which features contribute most to a target logit difference. ## Quick Start Load GPT-2 via TransformerLens, load a pretrained transcoder from the pchlenski/gpt2-transcoders HuggingFace weights, and run inference inside a TranscoderReplacementContext to inspect feature activations.
