perf-theory-tester

Validate performance hypotheses by applying a single change to a baseline, running at least two validation passes, and recording the delta. This structured workflow ensures results are repeatable and not skewed by workloads.

The single-change rule in performance benchmarking isolates variables by applying only one modification per experiment. This prevents overlapping effects, ensuring the measured delta accurately reflects the impact of the specific code or configuration tweak.

Run baseline comparisons by confirming the initial state, applying a single modification, and executing two or more validation passes. You then restore the baseline and record the hypothesis, change, delta, and verdict in a structured output.

Multiple validation passes are required for regression checks to eliminate noise and confirm repeatability. Running at least two passes ensures the measured performance delta is consistent and reliable under real workloads rather than an anomaly.

Feature-flag experiments can be tested across benchmarks by applying the flag as the single change. The workflow requires baseline confirmation, two or more validation passes, and a structured output capturing the hypothesis, delta, and final verdict.

The best way to document performance experiment results is using a standardized Output Format that captures the hypothesis, applied change, measured delta, final verdict, and supporting evidence. This ensures claims are repeatable and validated.