skills.rest
bowiemax
bowiemaxCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

ab-test-analysis

Analyze A/B test results for statistical significance and sample size validation.

Updated Mar 15, 2026
Search Tags:#conversion rate#experiment analysis#ab testing#statistical significance#lift#product decisions
One-click install
npx skills add https://github.com/bowiemax/pm-skills --skill ab-test-analysis-bowiemax
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test-analysis
Source: https://github.com/bowiemax/pm-skills/tree/main/pm-data-analytics/skills/ab-test-analysis
Command: npx skills add https://github.com/bowiemax/pm-skills --skill ab-test-analysis-bowiemax

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps product managers and data analysts rigorously evaluate A/B test results, ensuring decisions are based on statistical significance and practical impact, not just intuition.

Core Features & Use Cases

  • Statistical Significance Calculation: Determines if observed differences between variants are statistically significant.
  • Sample Size & Duration Validation: Checks if the experiment was run with adequate data and for a sufficient period.
  • Recommendation Engine: Provides clear, actionable recommendations (Ship, Extend, Stop) based on the analysis.
  • Use Case: After running an A/B test on a new feature's conversion rate, use this Skill to confirm if the variant truly outperformed the control and decide whether to roll it out.

Quick Start

Analyze the A/B test results for the new checkout flow experiment using the provided data.

Frequently Asked Questions about ab-test-analysis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I analyze A/B test results for statistical significance?▼

To analyze A/B test results for statistical significance, you calculate conversion rates, lift, p-values, and confidence intervals to determine if observed differences between variants are valid and not due to random chance.

What is the best way to validate sample size and duration for split test data?▼

Validating sample size and duration for split test data involves checking if the experiment ran long enough with adequate data to ensure the observed conversion rate lift is statistically significant before making product decisions.

How do I calculate lift and p-values for an A/B testing experiment?▼

Calculating lift and p-values for an A/B testing experiment requires statistical computations on variant conversion rates to measure the performance difference and confirm if the outcome is statistically significant.

Can I generate a Python script for raw A/B test data analysis?▼

Yes, you can generate optional Python scripts for raw data to perform statistical calculations, evaluate experiment outcomes, and validate conversion rate differences when analyzing A/B test results.

When should I extend, stop, or ship a variant based on experiment analysis?▼

You should ship, extend, or stop a variant based on experiment analysis by evaluating statistical significance, practical impact, and whether the test duration and sample size met the required thresholds for data-driven product decisions.

Related Skills

Explore More Data & Analytics Skills→
nazlicancaglarnazlicancaglar

attribution-analyst

Compare attribution models and analyze conversion paths across marketing channels.

Community
Advanced
jay6697117jay6697117

agency-experiment-tracker

Design, track, and analyze hypothesis-driven experiments with statistical rigor.

Community
Advanced
omeraltnomeraltn

agency-analytics-reporter

Transform raw business data into analytics reports with SQL and Python.

Community
Advanced
docaohieu2808docaohieu2808

ab-test-setup

Plan and implement A/B and multivariate tests with Python-based statistical analysis.

Community
Intermediate