statistical-analyst

A/B test sample size planning requires calculating the required observations based on expected effect size, desired statistical power, and significance level. This Skill performs power calculations to estimate exactly how many users are needed before launch to ensure conclusive results.

Determining statistical and practical significance involves running hypothesis tests like proportion or chi-square tests to evaluate p-values and effect sizes. This Skill analyzes control and treatment comparisons, translating uncertainty bounds into clear business impact decisions.

Effect size measures the magnitude of the difference between groups, while confidence intervals provide a range of values likely to contain the true effect. This Skill computes both to help teams validate conversion lifts and estimate uncertainty risks.

Hypothesis testing applies to categorical comparisons in marketing and product analytics through chi-square tests and proportion testing. This Skill supports these scenarios to help teams interpret significance and decide whether to ship, extend, or stop a test.

You should stop an A/B test when power analysis reveals the sample size is too small to detect a meaningful effect size. This Skill evaluates experimental risks and validates whether an experiment was sized well enough to be conclusive.