ab-test-analyzer

A/B test significance is determined by computing p-values, z-scores, and confidence intervals from per-variant visitors and conversions. This analysis compares a control against variants to validate if observed conversion rate differences are statistically robust.

Ship, extend, or kill an A/B test based on statistical power, significance, and sample size thresholds. Evaluating these metrics ensures the experiment has sufficient traffic to detect true lift and prevents premature decisions on underpowered variants.

Calculate required A/B test sample size by inputting daily traffic, confidence level, and test duration. This estimates remaining visitors needed per variant to achieve adequate statistical power and ensures the conversion experiment runs long enough to detect true lift.

Estimate revenue impact from A/B test conversion data by providing revenue per conversion alongside per-variant visitors and conversions. This calculates absolute and relative conversion lift to project the financial outcome of shipping a winning variant.

A/B testing validity checks for sample ratio mismatch compare actual visitor distribution across variants against expected traffic splits. Detecting this mismatch, enforcing minimum sample thresholds, and flagging novelty effects ensures statistical significance calculations remain unbiased.

Multiple comparison corrections are needed for A/B tests with multiple variants to prevent false positives. Applying these corrections adjusts p-values and confidence intervals when comparing one control against several variants, maintaining statistical significance integrity.