skills.rest
zjunlp
zjunlpOFFICIAL
Explore all skills by this author →
Website𝕏@ChenHuajunGitHub

neuron-value-weight

Analyze transformer feed-forward layers as key-value memories and compute key-value agreement.

75|7|Updated May 2, 2026
Search Tags:#transformer#mechanistic interpretability#neuron analysis#key-value memory#feed-forward layers#activation extraction#fairseq
One-click install
npx skills add https://github.com/zjunlp/Mechanist --skill neuron-value-weight-zjunlp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: neuron-value-weight
Source: https://github.com/zjunlp/Mechanist/tree/main/skills/mechanism-skills/vocabulary-projection/neuron-value-weight
Command: npx skills add https://github.com/zjunlp/Mechanist --skill neuron-value-weight-zjunlp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pandas, numpy, psutil, matplotlib, and includes scripts (resource) and references (resource) components.

What problem does it solve? Understanding what individual neurons in transformer feed-forward layers encode is difficult without tooling to extract activations, identify trigger examples, and measure how value vectors align with their keys. ## Core Features & Use Cases - Trigger Example Extraction: Identify the top input examples that activate specific keys (layer/dimension pairs) in transformer FFN layers, with JSONL-to-text conversion for readable inspection. - Prediction Extraction: Extract layer-level and dimension-level FFN predictions into pickle files, with analysis, CSV export, and mode comparison utilities. - Key-Value Agreement Computation: Measure how strongly value vectors correspond to their associated keys, with statistical summaries and per-layer visualization. - Use Case: A mechanistic interpretability researcher studying a WikiText-103 language model can extract trigger examples for specific dimensions, then compute key-value agreement to test whether FFN layers behave as key-value memories. ## Quick Start Run the trigger example extraction script in quick-demo mode against a fairseq transformer checkpoint to extract and inspect trigger examples for sample dimensions.

Frequently Asked Questions about neuron-value-weight

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract trigger examples from transformer feed-forward layers?▼

Run extract_trigger_examples.py with a fairseq model checkpoint and tokenized data file. It calls analysis/generate_outputs.py with --get_trigger_examples, producing a JSONL file that can be converted to readable per-key text files.

How to compute key-value agreement in transformer FFN layers?▼

Use compute_key_value_agreement.py with a model directory and a directory of textual trigger examples. It outputs TSV and JSON files with per-key agreement scores, plus optional per-layer visualization via matplotlib.

What is the difference between dim and layer extraction modes?▼

Dimension-level extraction records predictions per individual FFN dimension, giving fine-grained detail at higher memory and time cost. Layer-level extraction aggregates predictions per layer, running faster and using less memory.

How much memory does key-value agreement analysis require?▼

The full key-value agreement analysis requires approximately 150GB of RAM. The script checks available memory and supports creating a subset of trigger example files for testing on smaller machines.

Why does trigger example extraction fail with a file not found error?▼

Extraction fails when the model checkpoint path or tokenized data file is missing. Verify the model_dir points to a valid fairseq checkpoint and that WikiText-103 preprocessing completed before running the scripts.

Related Skills

Explore More Education & Research Skills→
OpenLAIROpenLAIR
1.0k

sentencepiece

Train BPE or Unigram tokenizers on raw text and output model and vocab files.

Official
Intermediate
hxk622hxk622
8

hypothesis-generation

Formulate testable scientific hypotheses and design validation experiments.

Community
Advanced
taylorsatulataylorsatula
42

MI Analysis Operational Reference

Automate MI analysis of prompts with region annotation, attention capture, and logit-lens projections.

Community
Advanced
qcmuuqcmuu

nnsight-remote-interpretability

Trace, save, and patch PyTorch model activations with optional NDIF remote execution.

Community
Advanced