sparse-autoencoder-training

You can train and analyze Sparse Autoencoders (SAEs) to decompose neural network activations into interpretable features, uncovering underlying concepts and monosemantic representations learned by language models using SAELens and TransformerLens.

Superposition in neural networks is when models store multiple features within single neurons. You analyze it by training Sparse Autoencoders to decompose activations and study how disentangled, monosemantic representations form within language models.

Yes, you need PyTorch, TransformerLens, and SAELens installed. These dependencies provide the necessary environment for model interaction, activation extraction, and executing the feature decomposition required to train Sparse Autoencoders effectively.

You load a pre-trained Sparse Autoencoder to encode model activations for your specific prompt. This process reveals which interpretable features, such as sentiment or specific entities, are most active and how they contribute to the output.

Sparse Autoencoders are used for mechanistic interpretability because neural networks often store multiple features in superposition. SAEs decompose these complex activations into disentangled, monosemantic representations, making distinct concepts identifiable and easier to study.