ab-testing

Sample size depends on your baseline conversion rate, minimum detectable effect, 95% significance level, and 80% statistical power. The Skill provides quick-reference tables by baseline rate and lift, plus links to calculators like Evan Miller's and Optimizely's.

Divide total required sample by daily traffic to estimate duration, but run at least one full week to capture day-of-week variation. Avoid running longer than 4-8 weeks due to novelty effects and external factors.

Stopping early after peeking inflates false positive rates because early significance can reflect regression to the mean or audience mix shifts. Pre-commit to your calculated sample size, or use sequential testing methods in tools like Optimizely or PostHog.

A/B tests compare two versions with one change, A/B/n tests multiple variants requiring roughly 1.5-2x more sample, and multivariate tests combine multiple elements simultaneously with dramatically higher traffic needs. Low-traffic sites should prefer sequential A/B tests.

Define one primary metric tied to business value for calling the test, secondary metrics to explain why the change worked, and guardrail metrics to catch harm. For example, a pricing test might track plan selection rate, time on page, and refund rate.

An inconclusive result usually means insufficient sample size or too small an effect to detect. Options include accepting a larger minimum detectable effect, testing a bolder variant, combining traffic across similar pages, or making the decision with qualitative data instead.