ablation-planner

Designs and runs ablation studies for ML experiments using an external LLM reviewer.

75|7|Updated May 2, 2026
One-click install
npx skills add https://github.com/zjunlp/Mechanist --skill ablation-planner-zjunlp
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: ablation-planner
Source: https://github.com/zjunlp/Mechanist/tree/main/skills/ablation-planner
Command: npx skills add https://github.com/zjunlp/Mechanist --skill ablation-planner-zjunlp

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Once main experiment results support a claim, researchers still need rigorous ablation studies that anticipate reviewer questions before paper submission, and designing them manually is slow and easy to bias. ## Core Features & Use Cases - Reviewer-Driven Ablation Design: An external LLM reviewer (via the llm-chat MCP) proposes component ablations, hyperparameter sensitivity tests, and design-choice comparisons from a reviewer's perspective. - Structured Ablation Plan: Normalizes reviewer output into prioritized tables with what each ablation tests, expected outcomes, run order, and GPU-hour estimates. - Feasibility Review and Execution: CC checks compute budget and code-change requirements, smoke-tests each ablation, runs them in order, and logs results in EXPERIMENT_LOG.md. - Use Case: After your mechanistic interpretability method passes result-to-claim validation, invoke this skill to generate a complete ablation plan, run config-only ablations first, and record negative results as findings. ## Quick Start Ask the agent to plan ablation studies for your validated method by describing the method and its confirmed claims.

Frequently Asked Questions about ablation-planner

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design ablation studies for my ML paper?

Provide your method description, components, current results, and claims, and an external LLM reviewer proposes prioritized ablations covering component removal, hyperparameter sensitivity, and design alternatives. Each ablation includes what it tests and expected outcomes if the component matters.

When should I run ablation planning in a research workflow?

Run it after main results pass result-to-claim validation with claim_supported set to yes or partial, or when a reviewer identifies missing ablations. Running it before claims are validated wastes compute on unsupported experiments.

How is the external reviewer model configured for ablation planning?

The reviewer model resolves from project .mcp.json, user ~/.claude/settings.json, or shell environment variables LLM_MODEL, LLM_BASE_URL, and LLM_API_KEY, in that priority order. If none provide a model, the skill aborts rather than guessing a default.

What happens if my compute budget cannot cover all proposed ablations?

The skill checks GPU budget during feasibility review and proposes cutting lower-priority ablations, then asks the external reviewer to re-prioritize. It never silently drops ablations, and config-only ablations run first since they are cheaper.

Does the ablation planner record negative results?

Yes, all ablation outcomes are logged in EXPERIMENT_LOG.md, including cases where removing a component had no effect. Negative results are treated as important findings and incorporated into the updated findings documentation.