skills.rest
eddielueng
eddieluengCOMMUNITY
Explore all skills by this author →
GitHub

sparse-autoencoder-training

Train and analyze Sparse Autoencoders with SAELens and PyTorch.

1|Updated Apr 14, 2026
Search Tags:#model evaluation#superposition#feature discovery#mechanistic interpretability#saelens#sparse autoencoder#feature steering
One-click install
npx skills add https://github.com/eddielueng/hermes-agent-zh --skill sparse-autoencoder-training-eddielueng
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: sparse-autoencoder-training
Source: https://github.com/eddielueng/hermes-agent-zh/tree/main/optional-skills/mlops/saelens
Command: npx skills add https://github.com/eddielueng/hermes-agent-zh --skill sparse-autoencoder-training-eddielueng

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires sae-lens>=6.0.0, transformer-lens>=2.0.0, torch>=2.0.0, and includes references (resource) components.

What problem does it solve?

Sparse Autoencoders (SAEs) help decompose dense neural network activations into sparse, more interpretable features, making mechanistic interpretability and feature discovery feasible instead of guesswork.

Core Features & Use Cases

  • Train and evaluate SAEs with SAELens: set up training configurations, run training loops, and track interpretability-focused metrics like L0, CE recovery, dead features, and reconstruction quality.
  • Load pre-trained SAEs and analyze features: encode activations into sparse features, inspect top-activating features per token/position, and quantify reconstruction performance.
  • Use features for interpretability workflows: perform feature attribution, steering with SAE directions, and causal feature ablation to test which features matter for specific predictions.

Quick Start

Train a custom Sparse Autoencoder on GPT-2 activations by instructing the AI to follow the Skill’s “Workflow 2: Training a Custom SAE” steps and fill in your chosen model hook point, dataset path, and SAE hyperparameters.

Frequently Asked Questions about sparse-autoencoder-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train a sparse autoencoder for mechanistic interpretability?▼

Train a sparse autoencoder for mechanistic interpretability by configuring training runs with SAELens to decompose neural network activations into sparse, interpretable features, tracking metrics like L0 and CE recovery across the training loop.

How do I analyze superposition and discover interpretable features in model activations?▼

Analyze superposition and discover interpretable features by loading pre-trained SAEs to encode dense activations into sparse features, then inspecting top-activating features per token or position to quantify reconstruction performance and reveal underlying feature representations.

Can I perform feature steering and causal ablation using SAELens with TransformerLens?▼

Yes, you can perform feature steering and causal ablation using SAELens with TransformerLens by applying SAE directions to steer model behavior or ablating specific sparse features to test their causal impact on specific predictions.

What metrics should I monitor when evaluating sparse autoencoder training quality?▼

Monitor L0 sparsity, cross-entropy loss recovery, dead feature counts, and reconstruction quality when evaluating sparse autoencoder training to ensure the model decomposes activations into interpretable features without losing predictive performance.

Do I need PyTorch and specific library versions to set up sparse autoencoder training?▼

Yes, you need PyTorch 2.0.0 or later, SAELens 6.0.0 or later, and TransformerLens 2.0.0 or later to configure SAE training architectures and execute training loops for decomposing neural network activations into sparse features.

Related Skills

Explore More Education & Research Skills→
ihatesea69ihatesea69
3

sparse-autoencoder-training

Train and analyze Sparse Autoencoders on neural network activations using SAELens, TransformerLens, and PyTorch.

Community
Advanced
cxnaivecxnaive

sparse-autoencoder-training

Train and analyze Sparse Autoencoders to decompose dense neural activations into sparse features.

Community
Advanced
founderphantomfounderphantom

sparse-autoencoder-training

Train and analyze Sparse Autoencoders with SAELens to decompose activations into interpretable features.

Community
Intermediate
ragnarokhaaragnarokhaa
4

sparse-autoencoder-training

Train Sparse Autoencoders to decompose neural network activations into interpretable features.

Community
Advanced