experiment-design

To design an A/B test, you define a hypothesis, pre-declare a primary metric, calculate the required sample size, set an observation window, and establish a decision rule before shipping the change. This ensures rigorous experiment design and measurement.

Hypothesis-first experiment planning anchors product decisions to evidence by requiring a pre-defined hypothesis and metrics before shipping changes. It prevents biased evaluation by forcing you to commit to how success is measured before observing results.

You calculate sample size and set an observation window by defining the target minimum detectable effect (MDE) and pre-declaring the primary metric. This structured experiment design process applies to A/B, multivariate, and before-after experiments across product surfaces.

Yes, this experiment design approach explicitly supports before-after experiments alongside A/B and multivariate tests. It requires a pre-declared primary metric, sample size calculation, observation window, and revert plan to ensure valid before-after comparisons.

An experiment plan should include the proposed change, hypothesis, primary metric, target MDE, observation window, and pre-declared decision rule. It must also define a clear revert pathway and log the planned experiment to experiment-log.md for centralized tracking.

Pre-declared decision rules are essential for rigorous experiment design and should be bypassed only for emergency shipping where immediate action is required. Without pre-commitment to metrics and sample sizes, post-hoc analysis risks invalidating your experiment results.