sparse-autoencoder-training

Train and analyze Sparse Autoencoders with SAELens on GPT-2 activations.

228k|44.7k|Updated Jul 22, 2025

Search Tags:#feature discovery #mechanistic interpretability #neural network analysis #saelens #sparse autoencoder #representation learning

Ready to run

npx skills add https://github.com/NousResearch/hermes-agent --skill sparse-autoencoder-training-nousresearch

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: sparse-autoencoder-training
Source: https://github.com/NousResearch/hermes-agent/tree/main/skills/mlops/saelens
Command: npx skills add https://github.com/NousResearch/hermes-agent --skill sparse-autoencoder-training-nousresearch

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires sae-lens, transformer-lens, torch, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the challenge of polysemanticity and superposition in neural networks by decomposing dense activations into sparse, interpretable features using Sparse Autoencoders (SAEs).

Core Features & Use Cases

Feature Discovery: Uncover interpretable features within model activations.
Mechanistic Interpretability: Analyze superposition and study feature geometry.
Use Case: When analyzing a language model's behavior, use this Skill to identify specific features that activate for concepts like "legal language" or "HTTP requests" in its activations, aiding in understanding its learned representations.

Quick Start

Use the sparse-autoencoder-training skill to train a custom SAE on GPT-2 small model activations.

Frequently Asked Questions about sparse-autoencoder-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I decompose neural network activations into interpretable features using sparse autoencoders?▼

To decompose neural network activations into interpretable features, train a sparse autoencoder using SAELens to isolate monosemantic representations. This process resolves polysemanticity by breaking dense activations into sparse, interpretable features for analysis.

What is superposition in mechanistic interpretability and how do I analyze it?▼

Superposition in mechanistic interpretability occurs when a neural network represents more features than it has dimensions by compressing dense activations. You analyze it by training a sparse autoencoder to decompose model activations and study the resulting feature geometry.

How do I train a custom sparse autoencoder on GPT-2 small model activations?▼

Train a custom sparse autoencoder on GPT-2 small model activations by using SAELens alongside transformer-lens and torch. This Skill provides the necessary scripts and guidance to hook into model activations and initiate SAE training for feature discovery.

Does SAELens work with Gemma models for feature discovery and steering?▼

Yes, SAELens works with Gemma models for feature discovery and feature-based steering. It facilitates identifying specific interpretable features within Gemma activations, allowing you to analyze learned representations and guide model behavior.

Can I use transformer-lens and torch to identify monosemantic representations in language models?▼

Yes, you can use transformer-lens and torch to identify monosemantic representations by training a sparse autoencoder on model activations. This combination enables you to extract interpretable features that activate for specific concepts like legal language or HTTP requests.

Why does my neural network exhibit polysemanticity and when do I need a sparse autoencoder?▼

Neural networks exhibit polysemanticity when individual neurons activate for multiple unrelated concepts due to superposition. You need a sparse autoencoder when analyzing language model behavior to untangle these dense activations into distinct, interpretable features.