auto-experiment

Implements, reviews, and deploys LLM interpretability experiments from an experiment plan.

75|7|Updated May 2, 2026
One-click install
npx skills add https://github.com/zjunlp/Mechanist --skill auto-experiment-zjunlp
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: auto-experiment
Source: https://github.com/zjunlp/Mechanist/tree/main/skills/auto-experiment
Command: npx skills add https://github.com/zjunlp/Mechanist --skill auto-experiment-zjunlp

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Turning a written experiment plan into running GPU experiments requires writing code, reviewing it for bugs, routing to the right mechanism family, and deploying runs — a slow, error-prone manual process. This Skill automates that bridge between idea discovery and the auto-review loop, producing initial results ready for iteration. ## Core Features & Use Cases - Plan-driven implementation: Parses EXPERIMENT_PLAN.md, EXPERIMENT_TRACKER.md, and FINAL_PROPOSAL.md to generate experiment code for sanity, baseline, main, and ablation milestones. - Mechanism-family routing: Routes the plan through mechanism-skills and experiment-tips catalogs to commit a mechanism family and adopt reproducibility conventions before any code is written. - Phenomenon-validation gate: Runs a phenomenon-validation milestone first with integrity auditing and a bounded hyperparameter-tuning retry budget before spending mechanism compute. - Cross-model code review and GPU deployment: Sends experiment code to an external LLM reviewer, then deploys runs in parallel with GPU partitioning, OOM auto-scale-up, and resume support. - Use Case: A mechanistic interpretability researcher finishes a refined proposal and says "implement experiments" — the Skill routes the mechanism family, writes and reviews the code, deploys runs across available GPUs, and collects initial results for the auto-iteration loop. ## Quick Start Tell the agent to implement and deploy the experiments from refine-logs/EXPERIMENT_PLAN.md for your research direction.

Frequently Asked Questions about auto-experiment

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I implement experiments from an experiment plan automatically?

Invoke the skill with a research direction after generating EXPERIMENT_PLAN.md and FINAL_PROPOSAL.md. It parses milestones, routes the mechanism family, generates code, runs a cross-model review, and deploys runs to GPUs, collecting initial results for the iteration loop.

What inputs does the auto-experiment workflow require?

It expects refine-logs/EXPERIMENT_PLAN.md as the primary input, plus EXPERIMENT_TRACKER.md and FINAL_PROPOSAL.md for context. If none exist, it asks the user what experiments to implement.

Can I resume an experiment run after a crash?

Yes. Set RESUME to true and each phase checks whether its primary artifact already exists and skips completed work, such as committed routing files, reviewed code, and runs marked done in the tracker. Resume never deletes pre-existing files.

How does it handle GPU out-of-memory errors during experiments?

On OOM it reduces batch size with gradient accumulation, enables gradient checkpointing, and automatically adds free GPUs with sharding after a numeric-equivalence check. Only when the GPU cap is reached does it halt and report options.

What happens if the phenomenon under study does not reproduce?

A phenomenon-validation gate runs the validation milestone first with an integrity audit. If the effect is not established, it retries with hyperparameter tuning within a five-iteration budget, then terminates with a negative-result report and a ledger entry.