ab-test-analysis

Name: ab-test-analysis
Availability: InStock
Author: osirs

Analyze A/B test results for statistical significance and sample size validation.

Updated Mar 8, 2026

Search Tags:#decision making #experimentation #a/b testing #product metrics #data interpretation #statistical analysis

Ready to run

npx skills add https://github.com/Osirs/pm-skills-sc --skill ab-test-analysis-osirs

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: ab-test-analysis
Source: https://github.com/Osirs/pm-skills-sc/tree/main/.trae/skills/ab-test-analysis
Command: npx skills add https://github.com/Osirs/pm-skills-sc --skill ab-test-analysis-osirs

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pandas, numpy, scipy, statsmodels, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill helps you rigorously analyze A/B test results, ensuring data-driven product decisions by evaluating statistical significance, sample size, and potential impacts on key metrics.

Core Features & Use Cases

Statistical Significance Testing: Determines if observed differences between variants are statistically meaningful.
Sample Size and Duration Validation: Checks if the test collected enough data for reliable conclusions.
Guardrail Metric Monitoring: Identifies any negative side effects on secondary metrics.
Decision Framework: Provides clear recommendations (Ship, Extend, Stop, Investigate) based on the analysis.
Use Case: You've run an A/B test on a new feature and need to know if the uplift in conversion rate is real and if it negatively impacted user engagement.

Quick Start

Analyze the A/B test results from the provided data file for the new checkout flow.

Frequently Asked Questions about ab-test-analysis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I determine if my A/B test results are statistically significant?▼

A/B test analysis evaluates statistical significance by measuring observed differences between variants using scipy and statsmodels, providing confidence intervals to validate whether the uplift is real.

How do I check if my A/B test collected enough data for reliable conclusions?▼

Sample size validation assesses whether your A/B test collected enough data by checking the duration and volume against the required threshold, confirming reliability before making product decisions.

What is the best way to monitor guardrail metrics during A/B testing?▼

Guardrail metric monitoring identifies negative side effects on secondary metrics during A/B testing, ensuring that changes in core metrics do not harm overall user engagement.

Can I use Python to get actionable recommendations from A/B test data?▼

Yes, this A/B test analysis processes data with pandas and numpy to generate actionable recommendations, providing a clear decision framework to ship, extend, stop, or investigate the test.

When do I need to extend or stop an A/B test instead of shipping it?▼

You should extend or stop an A/B test when the analysis shows insufficient sample size, lacks statistical significance, or detects negative impacts on guardrail metrics despite core metric changes.