experiment-design

To calculate sample size and duration for A/B testing, you must define primary and guardrail metrics first. This ensures your experiment has adequate statistical power to reach significance and avoids underpowered tests that produce inconclusive or misleading results.

Peeking in A/B testing is checking results before reaching the required sample size, which invalidates statistical significance. You can prevent this by establishing a pre-committed analysis plan that dictates exactly when and how to evaluate the experiment data.

Writing a clear hypothesis for a new product feature requires formulating an evidence-based, measurable prediction. It involves defining the expected change, selecting primary metrics to track, and setting guardrail metrics to catch any negative side effects during the A/B test.

Testing multiple variables simultaneously in experiment design causes issues by making it difficult to attribute observed metric changes to a specific variation. A pre-committed analysis plan helps structure tests to isolate variables and maintain statistical rigor.

Guardrail metrics are secondary metrics used in hypothesis testing to ensure a new feature does not cause negative side effects. You need them during experiment setup alongside primary metrics to monitor overall system health and prevent unintended regressions.