sparse-autoencoder-training

Name: sparse-autoencoder-training
Availability: InStock
Author: gregsgreycode

Train and analyze sparse autoencoders on transformer activations using SAELens.

6|2|Updated Mar 19, 2026

Search Tags:#feature discovery #mechanistic interpretability #sparse autoencoder #sae-lens #neural activations #monosemanticity

Ready to run

npx skills add https://github.com/GregsGreyCode/Logos --skill sparse-autoencoder-training-gregsgreycode

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: sparse-autoencoder-training
Source: https://github.com/GregsGreyCode/Logos/tree/main/skills/mlops/evaluation/saelens
Command: npx skills add https://github.com/GregsGreyCode/Logos --skill sparse-autoencoder-training-gregsgreycode

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Provides a guided approach to training and analyzing Sparse Autoencoders (SAEs) using SAELens to decompose Transformer activations into sparse, interpretable features for mechanistic interpretability.

Core Features & Use Cases

Train SAEs on model activations to expand and sparsify representations.
Analyze activations to discover interpretable, monosemantic features and study superposition.
Evaluate reconstruction quality and feature sparsity, enabling feature-based steering and ablation experiments.

Quick Start

Install sae-lens, load a pretrained SAE for your target model, and encode activations to inspect sparse features and reconstruction quality.

Frequently Asked Questions about sparse-autoencoder-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I train sparse autoencoders on transformer activations using SAELens?▼

To train sparse autoencoders with SAELens, configure the SAE architecture (d_in, d_sae, activation) and training hyperparameters like lr and l1_coefficient to sparsify model representations. It integrates with TransformerLens for end-to-end experimentation.

What is mechanistic interpretability and how do sparse autoencoders help?▼

Mechanistic interpretability studies model internals, and sparse autoencoders decompose transformer activations into interpretable, monosemantic features. Training SAEs expands dense representations into sparse, interpretable features to analyze how neural networks process information.

Can I use SAELens to evaluate sparse autoencoder feature sparsity and reconstruction quality?▼

Yes, SAELens supports evaluating sparse autoencoder reconstruction quality and feature sparsity. You can analyze trained SAEs across model layers to verify monosemanticity and prepare features for steering or ablation experiments.

Does sparse autoencoder training work with TransformerLens for end-to-end experimentation?▼

Sparse autoencoder training integrates with TransformerLens to enable end-to-end mechanistic interpretability experimentation. You can load pretrained SAEs for target models, encode activations, and inspect sparse features directly within the workflow.

What SAE training hyperparameters do I need to configure for feature discovery?▼

For feature discovery, configure SAE architecture parameters like d_in and d_sae, along with training hyperparameters including learning rate, l1_coefficient, and warm-up. These control feature sparsity and optimization during sparse autoencoder training.

When do I need sparse autoencoders to study superposition in neural networks?▼

You need sparse autoencoders to study superposition when analyzing how neural networks encode more features than dimensions. Training SAEs on transformer activations decomposes dense representations into sparse, monosemantic features for interpretability research.