ab-test-analyzer

What problem does it solve?

Removes guesswork from A/B testing by performing rigorous statistical analysis on variant results and producing a clear recommendation to ship, extend, or kill a test based on significance, power, and validity checks.

Core Features & Use Cases

• Comprehensive statistics: Calculates per-variant conversion rates, confidence intervals, pooled z-scores, p-values, absolute and relative lift, and statistical power.
• Validity and safety checks: Detects sample ratio mismatch, enforces minimum sample thresholds, flags novelty effects, and applies multiple comparison corrections when needed.
• Decision & impact analysis: Estimates required sample sizes and days remaining given traffic, computes revenue impact when revenue per conversion is provided, and issues actionable SHIP/EXTEND/KILL guidance with next steps.
• Use case: Decide whether to implement a new CTA, continue an underpowered experiment, or stop a losing variant while estimating business impact.

Quick Start

Analyze this A/B test and recommend SHIP, EXTEND, or KILL: Control 2500 visitors, 100 conversions; Variant B 2500 visitors, 150 conversions; use 95% confidence.