ab-test-analysis

Analyze A/B test results for statistical significance and actionable recommendations.

Updated Feb 6, 2026

Search Tags:#conversion rate #experiment analysis #ab testing #statistical significance #product analytics #a/b test

Ready to run

npx skills add https://github.com/spandios/vibe-utils --skill ab-test-analysis-spandios

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: ab-test-analysis
Source: https://github.com/spandios/vibe-utils/tree/main/skills/ab-test-analysis
Command: npx skills add https://github.com/spandios/vibe-utils --skill ab-test-analysis-spandios

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you rigorously analyze A/B test results, determine statistical significance, and make confident decisions about shipping product changes.

Core Features & Use Cases

Statistical Significance: Calculate p-values and confidence intervals to understand if observed differences are real.
Setup Validation: Check for adequate sample size, test duration, and potential biases like sample ratio mismatch.
Recommendation Engine: Provides clear recommendations (Ship, Extend, Stop) based on results and guardrail metrics.
Use Case: You've run an A/B test on a new feature and need to know if the uplift in conversion rate is statistically significant and if any other key metrics were negatively impacted.

Quick Start

Analyze the A/B test results for the new checkout flow using the provided data files.

Frequently Asked Questions about ab-test-analysis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I calculate statistical significance for my a/b test results?▼

To calculate statistical significance for a/b testing, you need to evaluate p-values and confidence intervals. This Skill performs those statistical calculations to determine if observed differences in your experiment data are real, validating whether your uplift is meaningful.

What is sample ratio mismatch and how do I check it in experiment analysis?▼

Sample ratio mismatch is a potential bias in a/b testing where traffic allocation skews from expected proportions. This Skill validates your test setup by checking for adequate sample size, test duration, and potential biases like sample ratio mismatch to ensure reliable experiment analysis.

How do I validate my test setup and sample size for an a/b test?▼

Validating test setup for a/b testing involves verifying adequate sample size, checking test duration, and monitoring guardrail metrics. This Skill evaluates these experiment parameters to ensure your product analytics data is robust enough for confident, data-driven decisions.

Can I get actionable recommendations on whether to ship or stop a product feature?▼

Yes, this Skill provides actionable recommendations to ship, extend, or stop a product feature. It evaluates your a/b test results and guardrail metrics to help you make confident, data-driven decisions about shipping product changes without negatively impacting key metrics.

Does this tool handle confidence interval calculation for conversion rate experiments?▼

Yes, this Skill handles confidence interval calculation for conversion rate experiments. It analyzes a/b test results by computing statistical significance and confidence intervals to help you understand if the observed differences in your experiment data are real.

When should I extend my a/b test instead of shipping the variant?▼

You should extend your a/b test when results lack statistical significance or sample size validation fails. This Skill provides clear recommendations to ship, extend, or stop based on your experiment analysis, p-values, and guardrail metric monitoring.