ab-test-setup

Start by locking a single hypothesis with a defined audience, primary metric, and Minimum Detectable Effect. Then calculate required sample size from baseline rate, significance level, and power, set guardrail metrics, and verify tracking before implementation.

A valid hypothesis includes an observation or evidence, one specific change, a directional expectation, a defined audience, and measurable success criteria. It must be explicitly confirmed and locked before designing variants or metrics.

Use multivariate testing only when you need to measure interaction effects between multiple changes and have very high traffic. Default to a simple A/B test for a single change, since MVT requires substantially larger sample sizes.

Stopping a test early because results look good inflates false positive rates and invalidates statistical conclusions. The Skill enforces committing to a pre-calculated duration and forbids stopping early or redefining success criteria mid-test.

Guardrail metrics are measurements that must not degrade during a test, such as page performance or revenue. A significant guardrail failure blocks shipping the variant even if the primary metric wins.

Refuse when the baseline rate is unknown, traffic is insufficient to detect the MDE, the primary metric is undefined, or multiple variables change without proper design. In these cases, delay or redesign the experiment first.