experiment-audit

What problem does it solve? LLM agents running experiments can silently produce fraudulent results — fake ground truth derived from model outputs, self-normalized scores inflated to 0.99, phantom numbers from nonexistent files, or tiny pilots reported as comprehensive evaluations. This Skill adds an independent integrity check that catches these failure modes before results reach a paper. ## Core Features & Use Cases - Six-Check Audit (A–F): Verifies ground-truth provenance, score normalization, result-file existence and number consistency, dead metric code, scope-vs-claims alignment, and evaluation-type classification. - Cross-Model Reviewer Independence: The executor only collects file paths; an external LLM reviewer (via llm-chat MCP) reads the code and issues the PASS/WARN/FAIL verdict, preventing self-grading. - Per-Claim Scoping: Each invocation audits one claim's linked milestones from EXPERIMENT_PLAN.md and EXPERIMENT_TRACKER.md, producing EXPERIMENT_AUDIT.md and EXPERIMENT_AUDIT.json. - Use Case: After running experiments for claim C1 of a mechanistic interpretability study, invoke the audit to confirm C1's evaluation used real dataset ground truth and that every reported number matches an actual result file. ## Quick Start Run the experiment audit on the refine-logs directory for claim C1 and write the report to the claim's audit output directory.