skills.rest
zjunlp
zjunlpOFFICIAL
Explore all skills by this author →
Website𝕏@ChenHuajunGitHub

inputs-and-layer-wise-states

Calculate and visualize layer-wise gradient norms in LLMs during fine-tuning.

75|7|Updated May 2, 2026
Search Tags:#llm#fine-tuning#visualization#pytorch#mechanistic interpretability#layer-wise#gradient analysis
One-click install
npx skills add https://github.com/zjunlp/Mechanist --skill inputs-and-layer-wise-states-zjunlp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: inputs-and-layer-wise-states
Source: https://github.com/zjunlp/Mechanist/tree/main/skills/mechanism-skills/gradient-detection/inputs-and-layer-wise-states
Command: npx skills add https://github.com/zjunlp/Mechanist --skill inputs-and-layer-wise-states-zjunlp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, numpy, transformers, matplotlib, seaborn, pandas, and includes scripts (resource) and references (resource) components.

What problem does it solve? Understanding how gradients flow through different layers of an LLM during fine-tuning is difficult without dedicated tooling, making it hard to compare training dynamics between fast and slow thinking tasks. ## Core Features & Use Cases - Gradient Calculation: Compute L2 gradient norms and SVD components for every model layer using Hugging Face Transformers and PyTorch. - Statistical Analysis: Aggregate gradient statistics including mean, standard deviation, coefficient of variation, and layer-section breakdowns (early, middle, late). - Visualization: Generate comparison plots, heatmaps, box plots, and violin plots contrasting fast vs slow thinking gradient patterns. - Use Case: Fine-tune a model on two datasets (fast vs slow reasoning responses), run gradient calculation on each, then produce side-by-side layer-wise comparison charts and a statistics table to identify which layers differ most. ## Quick Start Calculate layer-wise gradient statistics for my training data and visualize the differences between fast and slow thinking examples.

Frequently Asked Questions about inputs-and-layer-wise-states

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate per-layer gradient norms in PyTorch?▼

Run a forward and backward pass with labels set to the input ids, then iterate over model.named_parameters() and compute torch.norm(param.grad, p=2) for each parameter that has a gradient. The script saves results as JSONL per example.

How to compare gradients between fast and slow thinking fine-tuning?▼

Run gradient calculation separately on each dataset, then load both JSONL result files into the visualization script. It produces layer-wise comparison plots, heatmaps, and a statistics table with relative differences between the two conditions.

What input data format does gradient calculation require?▼

The script expects a JSON file containing a list of examples with either instruction and response fields or a single text field. Each example is tokenized with a configurable maximum sequence length, defaulting to 1024 tokens.

Does gradient analysis require a GPU?▼

The script loads models with device_map set to auto and float16 precision, so a CUDA GPU is the intended target. Processing full datasets layer by layer is memory-intensive, and the max_samples argument can limit workload size.

What are the limitations of layer-wise gradient norm analysis?▼

Gradient norms alone do not reveal causal mechanisms, only magnitude differences across layers. SVD analysis is skipped for one-dimensional parameters, and results depend heavily on the chosen dataset samples and sequence length truncation.

Related Skills

Explore More Education & Research Skills→
atrawogatrawog

reward

Train reward models with TRL RewardTrainer and LoRA for RLHF pipelines.

Community
Advanced
pur3v4d3rpur3v4d3r
6

hypothesis-generation

Generate testable scientific hypotheses with measurable predictions from observations and literature references.

Community
Advanced
Supporter09Supporter09

nnsight-remote-interpretability

Trace and manipulate PyTorch model internals for interpretability analysis.

Community
Advanced
MLT-OSSMLT-OSS
14

sparse-autoencoder-training

Train and analyze Sparse Autoencoders with SAELens to decompose neural network activations into interpretable features.

Official
Advanced