sparse-autoencoder-training

To decompose neural network activations into interpretable features, you train Sparse Autoencoders (SAEs) on those activations to learn monosemantic representations. This process reconstructs the dense activations into sparse features, enabling feature discovery and improving model interpretability.

A sparse autoencoder is used for neural network interpretability to break down dense model activations into sparse, interpretable features. By learning monosemantic representations, it allows you to perform feature discovery and understand which features predict specific tokens.

To train a sparse autoencoder using SAE Lens and Transformer Lens, you utilize the libraries to extract activations from a model like GPT-2. You then train the SAE on a specific layer, such as "blocks.8.hook_resid_pre", to reconstruct and analyze the learned interpretable features.

Yes, you can analyze which features predict specific tokens like 'Paris' in a language model. By training a sparse autoencoder, you can identify the interpretable features that are most predictive of certain tokens within their context, thus understanding the underlying concepts driving predictions.

Yes, you need PyTorch and specific libraries including SAE Lens and Transformer Lens to train sparse autoencoders for feature discovery. These dependencies provide the necessary framework to extract activations and learn the sparse, interpretable representations from neural networks.