suede-ab-testing

Sample size depends on your baseline conversion rate, minimum detectable effect, 95% significance, and 80% power. The Skill provides quick-reference tables: for example, a 3% baseline detecting a 20% lift needs about 31,000 visitors per variant, or use calculators like Evan Miller's or Optimizely's.

Run until three predeclared conditions are met: target sample per variant, minimum duration, and the decision rule. Minimum duration is at least one full week for day-of-week variation, two business cycles for B2B, and through paydays for e-commerce, regardless of early significance.

Checking results before reaching the predeclared sample size inflates false positive rates, a problem known as peeking. Early trends reverse routinely due to regression to the mean and audience mix shifts, so significance at partial sample is preliminary, not a winner.

Multivariate tests multiply combinations (2 headlines x 2 images x 2 CTAs = 8 variants) and require very high traffic. If your traffic cannot supply adequate sample per combination, run sequential A/B tests on individual elements instead.

Define three tiers: one primary metric tied to the hypothesis that decides the test, secondary metrics that explain why the change worked, and guardrail metrics that must not get worse, such as support tickets or refund rate.

Do not use it for analytics instrumentation (use suede-analytics), post-click conversion diagnosis (use suede-site-alchemy), or writing variant copy (use suede-copy). It also cannot help when traffic is too low to power any realistic test.