experimentation

To run an A/B test with rigorous analysis, define control and treatment groups, then apply frequentist or Bayesian methods, CUPED variance reduction, and sequential monitoring to generate a comprehensive HTML experiment report.

CUPED variance reduction is a statistical technique that uses pre-experiment covariates to reduce variance in A/B testing, increasing statistical power and enabling smaller required sample sizes for detecting the minimum detectable effect.

Yes, sequential testing uses always-valid confidence intervals and alpha-spending to monitor experiments continuously, allowing early stopping when sufficient evidence is gathered without inflating false positive error rates.

Sample size estimation, minimum detectable effect (MDE), and power analysis are calculated during the experiment design phase by specifying the desired statistical power, significance level, and expected baseline conversion rates.

Yes, the analysis supports both frequentist and Bayesian methods, allowing you to compute p-values and confidence intervals or posterior distributions and credible intervals depending on your preferred statistical framework.

Sample Ratio Mismatch (SRM) checks diagnose whether the actual traffic split between control and treatment deviates from the intended allocation, indicating critical data quality or instrumentation issues before interpreting experiment results.