ab-test-analysis

Use power analysis with scipy: compute Cohen's h effect size from baseline rate and minimum detectable effect, then apply z-scores for your alpha (typically 0.05) and power (typically 0.80). For a 10% baseline detecting a 2% lift, you need roughly 3,842 users per variant.

Run a two-proportion z-test: compute the pooled proportion, standard error, and z-score from conversions and visitors per variant, then derive a two-tailed p-value with scipy.stats.norm. Report the 95% confidence interval alongside the p-value.

Frequentist tests use z-tests and p-values to reject a null hypothesis at a fixed significance level. Bayesian testing with PyMC3 uses a Beta-Binomial model to directly estimate the probability that variant B beats A, plus a credible interval for the lift.

Checking results daily and stopping when significant inflates the false positive rate from 5% to roughly 30% due to repeated testing. The fix is fixed-horizon testing: decide sample size upfront and analyze only once when it is reached.

Apply a Bonferroni correction by dividing your significance level by the number of metrics tested. Testing 20 metrics at alpha 0.05 requires p < 0.0025 per test to avoid false positives from multiple comparisons.

Use hash-based assignment: hash the user ID combined with the test name using MD5, then take the modulo by the number of variants. This guarantees the same user always sees the same variant across sessions and devices without cookies.