What problem does it solve? Product teams often misread experiment data—calling winners too early, ignoring underpowered tests, or missing guardrail regressions. This Skill applies rigorous statistical methods to A/B test results and converts the findings into a clear launch, extend, or stop decision. ## Core Features & Use Cases - Statistical Significance Testing: Computes conversion rates, relative lift, p-values (z-test or chi-square), and 95% confidence intervals for control vs. variant groups. - Test Validity Checks: Verifies sample size adequacy (power analysis), test duration across business cycles, randomization integrity (SRM detection), and novelty effects. - Guardrail Metric Review: Flags cases where the primary metric wins but revenue, engagement, or performance guardrails degrade. - Decision Framework: Maps result patterns to concrete recommendations—ship, investigate, extend, or stop—with a structured markdown summary. - Use Case: You ran a two-week checkout button experiment with 50,000 users per arm. Provide the conversion counts, and the Skill validates power, computes significance, checks guardrails, and tells you whether to roll out the variant. ## Quick Start Analyze this A/B test result: control had 12,000 visitors with 480 conversions, variant had 12,000 visitors with 540 conversions, and tell me whether to ship it.
