skills.rest
Osirs
OsirsCOMMUNITY
Explore all skills by this author →
GitHub

ab-test-analysis

Analyze A/B test results for statistical significance and sample size validation.

Updated Mar 8, 2026
Search Tags:#decision making#experimentation#a/b testing#product metrics#data interpretation#statistical analysis
One-click install
npx skills add https://github.com/Osirs/pm-skills-sc --skill ab-test-analysis-osirs
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test-analysis
Source: https://github.com/Osirs/pm-skills-sc/tree/main/.trae/skills/ab-test-analysis
Command: npx skills add https://github.com/Osirs/pm-skills-sc --skill ab-test-analysis-osirs

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pandas, numpy, scipy, statsmodels, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill helps you rigorously analyze A/B test results, ensuring data-driven product decisions by evaluating statistical significance, sample size, and potential impacts on key metrics.

Core Features & Use Cases

  • Statistical Significance Testing: Determines if observed differences between variants are statistically meaningful.
  • Sample Size and Duration Validation: Checks if the test collected enough data for reliable conclusions.
  • Guardrail Metric Monitoring: Identifies any negative side effects on secondary metrics.
  • Decision Framework: Provides clear recommendations (Ship, Extend, Stop, Investigate) based on the analysis.
  • Use Case: You've run an A/B test on a new feature and need to know if the uplift in conversion rate is real and if it negatively impacted user engagement.

Quick Start

Analyze the A/B test results from the provided data file for the new checkout flow.

Frequently Asked Questions about ab-test-analysis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I determine if my A/B test results are statistically significant?▼

A/B test analysis evaluates statistical significance by measuring observed differences between variants using scipy and statsmodels, providing confidence intervals to validate whether the uplift is real.

How do I check if my A/B test collected enough data for reliable conclusions?▼

Sample size validation assesses whether your A/B test collected enough data by checking the duration and volume against the required threshold, confirming reliability before making product decisions.

What is the best way to monitor guardrail metrics during A/B testing?▼

Guardrail metric monitoring identifies negative side effects on secondary metrics during A/B testing, ensuring that changes in core metrics do not harm overall user engagement.

Can I use Python to get actionable recommendations from A/B test data?▼

Yes, this A/B test analysis processes data with pandas and numpy to generate actionable recommendations, providing a clear decision framework to ship, extend, stop, or investigate the test.

When do I need to extend or stop an A/B test instead of shipping it?▼

You should extend or stop an A/B test when the analysis shows insufficient sample size, lacks statistical significance, or detects negative impacts on guardrail metrics despite core metric changes.

Related Skills

Explore More Data & Analytics Skills→
andikarachmanandikarachman
14

data-preprocessing

Preprocess raw data into clean, analysis-ready datasets with pandas and numpy.

Community
Advanced
abhishekchoudhariabhishekchoudhari
1

ab-test-analysis

Analyze A/B test results for statistical and practical significance.

Community
Advanced
sunnypatneedisunnypatneedi
11

north-star-metrics

Define a North Star Metric and input metrics with owner assignments.

Community
Basic
jaechang-hitsjaechang-hits
298

pymc-bayesian-modeling

Perform Bayesian modeling workflows with PyMC and ArviZ.

Community
Advanced