What problem does it solve? Benchmarks, A/B tests, and model evaluations often contain hidden defects—prompt leakage, unequal arms, post-hoc analysis, or pooled results that hide failures—yet their reports look convincing. This Skill audits the raw artifacts behind any benchmark claim and produces a fail-closed verdict grounded in evidence rather than the author's conclusions. ## Core Features & Use Cases - Twelve hard validity gates: Scores preregistration, treatment isolation, prompt neutrality, leakage, oracle independence, grid integrity, exclusion handling, controls, variance, stratification, reproducibility, and claim scope as pass, fail, inconclusive, or not_applicable. - Fail-closed scoring script: Validates structured gate data with scripts/score_audit.py and derives the overall outcome so one failed hard gate overrides many passing checks. - Structured audit report: Generates a report from references/report-template.md covering recomputed effects, blockers with bias direction, claim-by-claim support, and residual uncertainty. - Use Case: A team claims their new prompting strategy beats the baseline in an A/B evaluation. Use this Skill to recompute per-stratum effects, check for answer leakage in model-visible prompts, verify the oracle rejects wrong answers, and determine whether the headline claim survives. ## Quick Start Use the audit-benchmark-evidence skill to audit this benchmark's raw results, prompts, and analysis code and grade whether its claims are supported.
