ab-testing-analyzer

Statistical significance testing determines whether observed differences between control and treatment groups are real or due to chance. This Skill applies t-tests, chi-squared tests, and Bayesian methods to experimental data, computing confidence intervals and effect sizes to validate conversion rate differences across your test groups.

Yes. The Skill segments experimental data by customer groups and performs retention and cohort analyses to reveal how different populations respond to treatments. Segmentation visualization dashboards show conversion lift and statistical significance within each segment.

Frequentist methods use fixed sample sizes and p-values; Bayesian methods quantify belief updates using prior distributions and posterior probabilities. This Skill supports both, letting you choose the framework that matches your experimental design and risk tolerance for early stopping or sequential testing.

Sample size planning predicts how many observations you need to detect a meaningful effect with statistical power. The Skill computes sample requirements based on baseline conversion, desired lift, significance level, and power, supporting informed experimental design before data collection begins.

Yes. The Skill generates exportable reports with interactive visualizations including retention curves and decision dashboards. Matplotlib and Seaborn produce publication-ready charts; reporting templates standardize output across multiple experiments and stakeholders.

Yes. When testing multiple hypotheses or segments simultaneously, multiple comparison corrections prevent false positives. The Skill applies these adjustments during hypothesis testing and supports multivariate AB designs to maintain statistical validity across all comparisons.