experiment-audit

To verify research reproducibility, you can audit experiment integrity by checking ground-truth provenance, metric normalization validity, and result-file existence against repository artifacts. This prevents overconfident claims by ensuring reported results are correctly computed and grounded.

To detect metric normalization fraud, audit your evaluation pipeline to flag suspicious score normalizations that divide by model-derived maxima, minima, or means. This ensures metrics are grounded in dataset values rather than manipulated model outputs.

To check if paper claims match actual result files, perform a result-to-claim consistency audit that verifies narrative performance claims reference real result files with matching metric keys and values. This validates that claims are supported by underlying artifacts.

Ground truth provenance verification confirms that evaluation targets are sourced from datasets rather than derived from model outputs without explicit proxy labeling. This maintains experiment integrity by validating the source of your evaluation benchmarks.

To audit experiment integrity for dead code and insufficient coverage, assess metric function call frequency and evaluation scope against claim strength. This warns when metric functions are never called or when coverage is too small for the claims made.

Yes, you can reduce bias when auditing model performance claims by separating collection from judgment using an executor to gather file paths and a separate read-only reviewer to produce an integrity verdict. This cross-model approach improves audit reliability.