sparse-autoencoder-training

To train a sparse autoencoder, you can use the SAELens integration provided here to configure pipelines via LanguageModelSAERunnerConfig and SAETrainingRunner. This workflow enables interpretable feature discovery by encoding dense model activations into sparse, monosemantic representations.

Evaluating sparse autoencoder reconstruction involves tracking metrics like L0 sparsity and cross-entropy (CE) recovery. The provided workflow documents these metrics to help you assess how well the SAE decomposes dense activations without losing essential language model representations.

Yes, you can load pre-trained SAEs using the SAELens workflow to encode activations and perform feature analysis. This allows you to identify interpretable features and apply targeted feature steering or ablation directly within your mechanistic interpretability research pipelines.

SAELens supports reproducible SAE training pipelines through configurable classes like LanguageModelSAERunnerConfig. This ensures consistent training and evaluation of sparse autoencoders, allowing researchers to reliably document L0 and CE recovery metrics during feature discovery.

Sparse autoencoders decompose dense model activations into sparse, interpretable features. This process improves mechanistic interpretability by isolating monosemantic representations, enabling researchers to study specific features and perform targeted analysis of language model behaviors.