ab-testing

Sample size depends on your baseline conversion rate, minimum detectable effect, significance level (usually 95%), and power (usually 80%). Use the quick reference tables in the sample size guide or online calculators like Evan Miller's to find the required visitors per variant.

Run until you reach your pre-calculated sample size, with a minimum of one full week to capture day-of-week variation. Avoid running longer than 4-8 weeks due to novelty effects and external factors, and never stop early based on peeking at results.

No. Stopping early based on peeked results inflates false positive rates because early significance can reflect regression to the mean or audience mix shifts. If you need early-stopping flexibility, use sequential testing methods supported by tools like Optimizely or VWO.

A/B testing compares two versions with a single change, while multivariate testing (MVT) tests combinations of multiple elements simultaneously. MVT requires dramatically more traffic since combinations multiply, so sequential A/B tests are often better for lower-traffic sites.

Define three tiers: a primary metric tied directly to your hypothesis, secondary metrics that explain why the change worked, and guardrail metrics that should not get worse. This prevents wins on one metric from hiding harm elsewhere.

Skip testing when traffic cannot reach the required sample size within a reasonable duration, when the change is too small to detect, or when the risk of a bad variant is unacceptable. In these cases, rely on qualitative research or test upstream where traffic is higher.