skills.rest
zjunlp
zjunlpOFFICIAL
Explore all skills by this author →
Website𝕏@ChenHuajunGitHub

sae-feature-activation-state

Analyze Sparse Autoencoder features through ablation, attribution, and k-sparse probing experiments.

75|7|Updated May 2, 2026
Search Tags:#attribution#mechanistic interpretability#sparse autoencoder#feature ablation#k-sparse probing#gemma-2#feature splitting
One-click install
npx skills add https://github.com/zjunlp/Mechanist --skill sae-feature-activation-state-zjunlp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: sae-feature-activation-state
Source: https://github.com/zjunlp/Mechanist/tree/main/skills/mechanism-skills/probing/sae-feature-activation-state
Command: npx skills add https://github.com/zjunlp/Mechanist --skill sae-feature-activation-state-zjunlp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, numpy, matplotlib, sae-spelling, and includes scripts (resource) and references (resource) components.

What problem does it solve? Understanding how Sparse Autoencoder (SAE) latents influence language model outputs requires running specialized experiments like feature ablation, attribution patching, and k-sparse probing, which are tedious to implement from scratch. ## Core Features & Use Cases - Feature Ablation: Ablate individual SAE features and measure their effect size, firing rate, and importance on model outputs for tasks like first-letter spelling. - Feature Attribution: Compute integrated gradient attribution scores to identify which SAE features contribute most to specific token predictions, with visualization support. - K-Sparse Probing: Train k-sparse probes across multiple k values to detect feature splitting by measuring how accuracy scales with the number of features used. - Use Case: Given a Gemma-2 model with a GemmaScope SAE at layer 12, run the ablation example to rank the most important features for spelling tasks, then use k-sparse probing to check whether the relevant information is split across multiple latents. ## Quick Start Run the feature ablation example script against a Gemma-2 model and GemmaScope SAE to identify the most important SAE features for a spelling task.

Frequently Asked Questions about sae-feature-activation-state

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I ablate individual SAE features and measure their impact?▼

Use the feature ablation example script, which loads a Gemma-2 model and GemmaScope SAE, then ablates individual features on spelling prompts. It reports effect size, firing rate, and importance scores per feature and saves aggregated results to JSON.

How to detect feature splitting in Sparse Autoencoders?▼

Train k-sparse probes across increasing k values and measure classification accuracy on SAE activations. If accuracy keeps improving as more features are added, the information is likely split across multiple latents rather than concentrated in one.

What is integrated gradient attribution for SAE features?▼

Integrated gradient attribution interpolates between a baseline input and the actual input, accumulating gradients at each step to score how much each SAE feature contributes to a target token prediction. The attribution example script compares this method against standard attribution.

Which models and SAEs does this analysis support?▼

The examples target Gemma-2 models loaded via TransformerLens-style hooks and GemmaScope SAEs configured by layer, width, and L0 sparsity. The sae_spelling library provides loaders for these specific configurations.

What are the limitations of k-sparse probing for feature analysis?▼

K-sparse probing requires labeled task data and enough training samples to fit probes reliably. Results depend on the chosen k values and regularization strength, and synthetic or small datasets may not reveal genuine splitting patterns.

Related Skills

Explore More Education & Research Skills→
twjarviszyra-webtwjarviszyra-web
3

sparse-autoencoder-training

Train and analyze sparse autoencoders for neural network activations.

Community
Advanced
0h-n00h-n0

research.attention.probe

Sets up TransformerLens and nnsight intervention experiments for mechanistic interpretability analysis of LLMs.

Community
Advanced
jonnabiojonnabio
6

pyvene-interventions

Performs customizable causal interventions on PyTorch neural network models.

Community
Advanced
Z43LZ43L

sparse-autoencoder-training

Train and analyze Sparse Autoencoders with SAELens to decompose neural activations into interpretable features.

Community
Advanced