saelens

Use LanguageModelSAERunnerConfig with a nested StandardTrainingSAEConfig specifying d_in, d_sae, and l1_coefficient, then run LanguageModelSAETrainingRunner. Enable l1_warm_up_steps around 1000 to prevent early feature death and monitor L0 and CE loss score.

Call SAE.from_pretrained with a release and sae_id, such as gpt2-small-res-jb and blocks.8.hook_resid_pre. In v6 this returns just the SAE object; use from_pretrained_with_cfg_and_sparsity if you also need the config dict and sparsity.

Version 6 uses nested configs: SAE options like d_sae and l1_coefficient move into a sae sub-config such as StandardTrainingSAEConfig, and logging moves into a LoggingConfig. Legacy flat options like hook_layer, architecture, and use_ghost_grads were removed.

Dead features usually result from applying the full L1 penalty too early. Set l1_warm_up_steps to around 1000 so the sparsity penalty ramps up gradually, and consider lowering l1_coefficient if reconstruction quality also suffers.

Use TransformerLens directly for basic activation analysis or causal intervention experiments that do not require learned sparse features. SAELens is appropriate when you need to discover interpretable features, study superposition, or perform feature-based steering.

Extract the feature direction from sae.W_dec at the desired feature index, then register a forward hook on the target hook point that adds the scaled direction to the residual stream during model.generate.