ab-test-analysis

To analyze A/B test results for statistical significance, this Skill calculates p-values and confidence intervals to determine if observed conversion rate differences are real. It validates whether your split test data supports shipping the variant.

The best way to decide whether to ship, extend, or stop an experiment is using a decision framework based on statistical significance, sample size validation, and guardrail metrics. This Skill evaluates experiment outcomes to provide actionable recommendations.

To check if your split test had enough participants, this Skill validates sample size and experiment duration against the observed conversion rates. This ensures the experiment results are reliable before making product decisions.

To monitor guardrail metrics during experiment analysis, this Skill checks that improvements in the primary conversion rate do not negatively impact other critical metrics. It ensures your product analytics reflect true overall improvement.

You need hypothesis testing for product analytics when evaluating split test data to distinguish real performance changes from random noise. This Skill applies statistical significance testing to validate your experiment outcomes.