skills.rest
DaddyElonMusk69
DaddyElonMusk69COMMUNITY
Explore all skills by this author →
GitHub

sparse-autoencoder-training

Train sparse autoencoders on transformer activations with SAELens.

Updated Apr 12, 2026
Search Tags:#training#model-interpretation#mechanistic-interpretability#sae#transformer-lens#sae-lens#sparse-autoencoder
One-click install
npx skills add https://github.com/DaddyElonMusk69/motis-agent --skill sparse-autoencoder-training-daddyelonmusk69
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: sparse-autoencoder-training
Source: https://github.com/DaddyElonMusk69/motis-agent/tree/main/optional-skills/mlops/saelens
Command: npx skills add https://github.com/DaddyElonMusk69/motis-agent --skill sparse-autoencoder-training-daddyelonmusk69

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Sparse Autoencoders (SAEs) provide a tractable path to decompose dense neural activations into sparse, interpretable features, enabling mechanistic interpretability research and feature discovery in transformers. This Skill guides researchers through loading pre-trained SAEs, configuring training runs, and analyzing activation sparsity and reconstruction quality.

Core Features & Use Cases

  • Step-by-step workflows for loading, training, and evaluating SAEs using SAELens and TransformerLens.
  • Guidance on architecture choices (standard, gated, top-k, jumprelu), sparsity control (L1 regularization, warm-up, ghost grads), and evaluation metrics (L0, CE recovery, dead features).
  • Real-world use cases include discovering interpretable features related to model activations, comparing features across prompts, and steering or ablation experiments for mechanistic studies.

Quick Start

Run a guided workflow to train a Sparse Autoencoder on GPT-2 resid_pre activations.

Frequently Asked Questions about sparse-autoencoder-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train sparse autoencoders on transformer activations?▼

Train sparse autoencoders on transformer activations by loading a model with TransformerLens, specifying a hook point and release, and configuring a training run in SAELens with L1 regularization to extract interpretable features.

What is the best way to control sparsity when training a sparse autoencoder?▼

Control sparsity when training a sparse autoencoder using techniques such as L1 regularization, warm-up schedules, and ghost grads, while evaluating the results with metrics like L0 and the percentage of dead features to ensure reconstruction quality.

Which sparse autoencoder architectures can I use for mechanistic interpretability?▼

For mechanistic interpretability, you can configure standard, gated, top-k, and jumprelu sparse autoencoder architectures to decompose dense neural activations into sparse, interpretable features.

Do I need TransformerLens to load pre-trained SAEs for feature discovery?▼

Yes, you need TransformerLens and SAELens to load pre-trained SAEs for feature discovery, as these frameworks provide the required workflows to analyze activation sparsity and evaluate CE recovery.

How does evaluating dead features improve sparse autoencoder training?▼

Evaluating dead features during sparse autoencoder training identifies inactive neurons that waste capacity and degrade reconstruction quality, allowing you to adjust sparsity controls and architecture choices to improve overall interpretability.

Related Skills

Explore More Education & Research Skills→
GregsGreyCodeGregsGreyCode
6

sparse-autoencoder-training

Train and analyze sparse autoencoders on transformer activations using SAELens.

Community
Advanced
cxnaivecxnaive

sparse-autoencoder-training

Train and analyze Sparse Autoencoders to decompose dense neural activations into sparse features.

Community
Advanced
afel6afel6

sparse-autoencoder-training

Train and analyze Sparse Autoencoders on transformer activations using SAELens.

Community
Advanced
InverterNetworkInverterNetwork

sparse-autoencoder-training

Trains and analyzes Sparse Autoencoders to decompose dense neural activations into sparse features.

Official
Advanced