ab-test-setup

A/B test sample size depends on your baseline conversion rate, minimum detectable effect, and statistical significance threshold. This skill guides you through calculating required sample sizes and test duration to ensure your experiment has sufficient power to detect meaningful improvements reliably.

A/B tests compare two variants; A/B/n tests evaluate multiple variants simultaneously; multivariate testing (MVT) isolates the impact of individual elements. This skill recommends which approach fits your hypothesis and provides sample size planning for each test type.

A strong hypothesis specifies what you'll change, why you expect improvement, and what metric proves success. This skill provides templates and frameworks to structure hypothesis-driven experiments so you measure impact consistently and avoid bias in result interpretation.

Effective experiments measure primary metrics (your success goal), secondary metrics (related outcomes), and guardrail metrics (to prevent unintended harm). This skill helps you define and document each metric type so results are trustworthy and actionable across stakeholders.

Results reporting requires documenting test duration, sample sizes, statistical significance, and confidence intervals alongside business impact. This skill provides templates and guidance to communicate findings clearly so teams make data-driven decisions on whether to ship changes.

Statistical significance prevents false positives from random variation, but business context matters too. This skill covers significance thresholds, practical significance, and guardrail metrics so you balance rigor with business constraints when deciding to implement changes.