ab-test-analysis

To calculate statistical significance for A/B testing, you need to analyze p-values and confidence intervals to determine if observed differences in conversion rates are meaningful rather than random noise. This Skill evaluates primary and guardrail metrics to validate if your test results reach statistical significance.

Validating A/B test setup involves assessing sample size adequacy, test duration, and potential biases like Sample Ratio Mismatch (SRM) or novelty effects. This Skill checks these experiment validity parameters to ensure your split test data is reliable before making product decisions.

Deciding whether to ship, stop, or extend an A/B test requires a structured decision framework based on primary and guardrail metrics. This Skill provides actionable recommendations by evaluating your test's lift, p-values, and statistical significance against experiment validity criteria.

Yes, you can analyze conversion rate optimization metrics for a new user onboarding flow by providing the split test data. The analysis calculates lift and p-values to determine if the onboarding changes produced statistically significant improvements in your primary metrics.

High lift without statistical significance in an A/B test often indicates insufficient sample size or short test duration, meaning the observed conversion rate differences could be random variance. This Skill assesses experiment validity and sample size adequacy to diagnose such issues.