meta-experiment

A/B testing with statistical significance involves designing controlled experiments that compare variants, collecting metrics, and analyzing results against a p < 0.05 threshold. This Skill automates the full lifecycle—from hypothesis creation through deployment decisions—ensuring minimum sample sizes and rigorous statistical validation before rollout.

Use controlled experiments to measure the impact of changes against control groups. This Skill analyzes experimental results with statistical rigor, automatically promoting changes when they meet significance thresholds or triggering rollback when regression is detected, removing guesswork from deployment decisions.

Both workflows are supported. You can manually run commands to check experiment status and analyze results, or enable auto-management to track metrics, enforce safety thresholds, and automatically promote or rollback based on statistical outcomes without manual intervention.

This Skill enforces minimum sample size requirements, statistical significance thresholds (p < 0.05), and automatic rollback when metrics show regression. These safeguards ensure only validated changes reach production and problematic variants are caught before broad deployment.

Proper experimental design and adequate sample size determine whether statistical results are reliable. This Skill requires you to specify both upfront, enforcing minimums to prevent underpowered experiments that might miss real regressions or produce false positives.