ab-test-setup

Sample size depends on your baseline conversion rate, minimum detectable effect, significance level (typically 95%), and statistical power (typically 80%). For example, detecting a 20% lift on a 3% baseline requires about 12,000 visitors per variant.

Run tests for at least one to two full business cycles, usually one to two weeks, and until the pre-calculated sample size is reached. Avoid running too long to prevent novelty effects and external factors from skewing results.

A/B testing compares two versions with a single change, while multivariate testing evaluates multiple changes in combination to detect interactions. MVT requires significantly more traffic and more complex analysis.

Stopping a test early when results look significant inflates false positive rates and exaggerates effect sizes. Pre-commit to a sample size, or use sequential testing methods if you must monitor results continuously.

Client-side tools like Optimizely or VWO work well for marketing pages and copy changes but may cause flicker. Server-side testing suits product features and performance-sensitive pages but requires development work.