ab-test-analysis

Analyze A/B test results for statistical significance and decision recommendations.

25.0k|2.7k|Updated Mar 1, 2026

Search Tags:#decision making #conversion rate #product metrics #experiment analysis #ab testing #statistical significance

Ready to run

npx skills add https://github.com/phuryn/pm-skills --skill ab-test-analysis-phuryn

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: ab-test-analysis
Source: https://github.com/phuryn/pm-skills/tree/main/pm-data-analytics/skills/ab-test-analysis
Command: npx skills add https://github.com/phuryn/pm-skills --skill ab-test-analysis-phuryn

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps product managers and data analysts rigorously evaluate A/B test results, ensuring decisions are based on statistically sound evidence and clear business impact.

Core Features & Use Cases

Statistical Significance: Determines if observed differences are likely due to the change or random chance.
Sample Size & Duration Validation: Checks if the test was run long enough with sufficient data.
Guardrail Metric Monitoring: Assesses the impact on secondary metrics to avoid unintended negative consequences.
Decision Recommendations: Provides clear "Ship," "Extend," "Stop," or "Investigate" recommendations with reasoning.
Use Case: After running an A/B test on a new feature, use this Skill to confirm if the increase in conversion rate is statistically significant and if any negative impacts on user engagement were observed, guiding the decision to launch or iterate.

Quick Start

Analyze the A/B test results for the new checkout flow using the provided data.

Frequently Asked Questions about ab-test-analysis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I analyze A/B test results for statistical significance?▼

To analyze A/B test results for statistical significance, the Skill evaluates whether observed differences in conversion rates are due to the product change or random chance, validating sample size and test duration.

What is the best way to evaluate guardrail metrics during experiment analysis?▼

Evaluating guardrail metrics during experiment analysis involves assessing the impact on secondary metrics to identify unintended negative consequences, ensuring that product iterations do not harm overall user engagement.

How do I know if my split test has enough sample size and duration?▼

To know if your split test has enough sample size and duration, the Skill validates the collected data against required thresholds, ensuring the experiment ran long enough to yield data-driven insights.

Can I get actionable recommendations on whether to ship or stop an A/B test?▼

Yes, you can get actionable recommendations to ship, extend, stop, or investigate experiments based on your A/B test data, providing clear reasoning for product iterations and decision making.

When should I not use an automated tool for A/B testing analysis?▼

You should not use an automated tool for A/B testing analysis when tests lack sufficient sample size or duration, as validating these constraints is required to accurately determine statistical significance.