experiment-audit

An experiment integrity audit verifies ground truth provenance, score normalization validity, and result file existence to prevent overconfident research claims. It generates structured PASS/WARN/FAIL findings and an audit report classifying claims as supported, qualified, or integrity-concerned.

Experiment integrity verification validates ground truth origins, detects normalization fraud, and confirms result file existence. You need to audit benchmark results after a run finishes and before writing narrative reports or paper sections to ensure reported numbers are evidenced by evaluated artifacts.

Detect dead metric code and score normalization fraud by running a cross-model integrity review where an executor collects artifact paths and a reviewer reads files to validate metrics. The audit flags suspicious normalization denominators and unused metric code in the output report.

The audit workflow requires completed experiment folders containing metrics, logs, configs, and paper claims. It operates through a two-stage process where an executor collects file paths and a reviewer reads files to produce structured JSON and markdown audit reports.

The best way to prevent scope inflation and phantom results is to run an integrity audit confirming result file existence, validating claimed values, and checking scope sufficiency. It produces a machine-readable JSON file to tag claims as supported, qualified, or integrity-concerned.

A key limitation is that the audit applies only to completed experiment folders where metrics, logs, configs, and paper claims already exist. It cannot audit incomplete runs or generate missing artifacts, requiring a two-stage workflow with separate executor and reviewer roles.