experiment-metrics

To select reliable A/B testing metrics, evaluate candidates using the STEDII framework criteria: Sensitive, Timely, Efficient, Debuggable, Interpretable, and Isolated. This structured approach ensures your chosen metrics accurately reflect feature impact and drive sound, data-driven decisions.

The STEDII framework is a structured evaluation method for experiment metric selection. It assesses potential metrics across six criteria—Sensitive, Timely, Efficient, Debuggable, Interpretable, and Isolated—to ensure you choose valid primary and guardrail metrics that provide actionable insights within your experiment window.

Brainstorm and score candidate metrics for an A/B test by generating a list of potential indicators, then evaluating each metric against the STEDII criteria. This scoring process helps identify a primary success metric and crucial guardrail metrics to protect against unintended regressions.

Pre-experiment checks needed before launching an A/B test include running A/A tests and performing sample ratio checks. These validation steps verify your experiment setup is sound, preventing metric selection pitfalls and ensuring the data collected will be reliable for analysis.

Common pitfalls when choosing guardrail metrics and primary success metrics include selecting indicators that are not timely or sensitive enough for the experiment window. Using the STEDII framework helps avoid these errors by validating metric isolation, interpretability, and efficiency before launch.