craft-experiment-design

To design an A/B test, formulate a falsifiable hypothesis stating the expected outcome and rationale, then define a single primary success metric, supporting secondary metrics, and guardrail metrics to prevent unintended degradation.

Guardrail metrics in experiment design are critical health indicators that must not degrade, such as error rates, system latency, and user retention, ensuring your product changes do not cause unintended negative side effects.

Audience allocation and test duration are determined by identifying the target audience, recommending a percentage traffic split, and estimating the runtime based on statistical assumptions needed to reach a valid conclusion.

A holdout strategy is used after an experiment concludes to maintain a control group, allowing you to measure the long-term impact and cumulative effect of shipping a feature compared to the baseline audience.

A robust experiment plan includes a falsifiable hypothesis, primary and secondary metrics, guardrails, audience allocation, holdout decision, duration estimate, and risk considerations, providing a complete structure for team review.

When planning feature experiments, consider risks and factors that could bias results, such as incorrect audience allocation, insufficient test duration, or degradations in guardrail metrics like latency and error rates.