ab-test-analysis

To analyze A/B test data for statistical significance, you input variant data and specify the primary metric to compute p-values, z-scores, and confidence intervals. This determines if observed variant differences are real and practically meaningful for shipping decisions.

A sample ratio mismatch (SRM) occurs when traffic allocation between variants deviates from the expected ratio. You can run SRM checks by inputting variant data into an A/B test analysis to detect mismatches and evaluate impacts on secondary guardrail metrics.

Yes, you can estimate required sample sizes and achieved power for experiment design by performing power analysis. This helps inform your A/B test study design and ensures you have sufficient data to detect meaningful effects before running the test.

To get a ship or don't ship recommendation from experiment results, the analysis evaluates statistical significance, confidence intervals, effect sizes, and guardrail impacts. It then generates a structured decision report with estimated business impact and clear guidance.

You need guardrail checks when evaluating A/B test variants to ensure that winning primary metrics do not negatively impact key secondary metrics. This prevents launching changes that could harm overall business health or user experience.

Yes, A/B test analysis works for evaluating email campaign open rates. You input the variant open rate data and design details, and it computes the statistical significance and effect sizes to help you decide whether to roll out to all users.