differential-audit

Freeze data hashes, package versions, seeds, and tolerances first, then compare every output channel — estimates, standard errors, warnings, and error statuses — not just headline numbers. Test the comparator itself by feeding it known-wrong values to confirm it fails rather than skips.

Classify outputs into binding tolerance classes: EXACT for names, counts, and statuses; stated absolute and relative tolerances for scalar numerics; and prespecified error-rate criteria for stochastic outputs. Never widen a tolerance silently — record it as an approved divergence with a reason.

Comparators often silently skip results they cannot reconcile, turning every green result into noise. Vaccinate the harness by feeding it a wrong value, a missing result, a misaligned row, and an empty result — it must fail on each, not skip.

No. Agreement only proves both implementations satisfy the same prespecified contract; both can be wrong in the same way, especially if one was written by reading the other. State explicitly that the audit establishes conformance, not correctness of the method.

Include analytically solvable known-truth cases, dirty-data problems like missing values and near-collinearity, compound cases combining several problems, published examples, and randomized valid designs. Have an independent reviewer add unseen fixtures to avoid testing only the implementer's mental model.