What problem does it solve?
This Skill prevents invalid or misleading marketing experiments by ensuring hypothesis-driven design, correct sample size calculation, appropriate test duration, and robust statistical analysis so decisions are based on reliable evidence rather than noisy results.
Core Features & Use Cases
- Hypothesis formation: Structured hypothesis templates that tie changes to measurable metrics and expected relative improvements.
- Test architecture guidance: Recommends A/B, A/B/n, multivariate, redirect, and bandit approaches and when to use each.
- Sample size & duration: Calculates required sample per variation using baseline conversion, MDE, significance, and power, and enforces minimum duration to avoid early stopping.
- Run rules & safeguards: Provides test rules (no peeking, one variable at a time, exclude anomalies) and segment analysis guidance to prevent false positives.
- Analysis & decision framework: Produces result summaries with p-values or Bayesian probabilities, confidence intervals, power achieved, winner determination, recommended actions, and follow-up tests.
- Use Case: Plan and analyze an A/B test comparing two landing page CTAs, estimate required traffic and duration, and produce a clear decision and next-step recommendation.
Quick Start
Create an A/B test plan comparing the current CTA text with a new value by defining the hypothesis, baseline CVR, minimum detectable effect, required sample size per variation, estimated duration, primary metrics, and success criteria.
