sparse-autoencoder-training

To decompose neural network activations into interpretable features, you train Sparse Autoencoders (SAEs) that map dense residual stream activations into sparse, monosemantic directions. This process isolates individual concepts like sentiment or legal language for targeted analysis.

Neurons activate for multiple unrelated concepts due to polysemanticity, a phenomenon where models superimpose many features within a single neuron. Sparse Autoencoders resolve this by decomposing dense activations into sparse, monosemantic features for mechanistic interpretability.

To identify top activating features using transformer-lens, you load a pre-trained Sparse Autoencoder and pass input text through the model. The SAE analyzes the residual stream activations to rank and extract the top features firing on that specific text.

Yes, you can use sparse autoencoders for model steering and causal ablation experiments. By isolating specific monosemantic features in the residual stream, you can target and modify or remove individual concepts to observe causal effects on model behavior.

Yes, you need sae-lens, transformer-lens, and torch to perform mechanistic interpretability analysis. These libraries provide the necessary functions to load models, train SAEs, and execute high-dimensional activation analysis and reconstruction workflows.

The best way to analyze feature geometry and superposition in neural networks is by training Sparse Autoencoders to decompose dense activations. This reveals the underlying sparse feature structure, allowing you to map safety-relevant concepts like bias or deception.