skills.rest
omeragaakbas
omeragaakbasCOMMUNITY
Explore all skills by this author →
GitHub

ab-test-analysis

Analyze A/B test results to compute statistical significance and practical impact.

Updated Mar 30, 2026
Search Tags:#python#experiment-analysis#ab-testing#sample-size#power-analysis#statistical-significance#confidence-interval
One-click install
npx skills add https://github.com/omeragaakbas/zoyare --skill ab-test-analysis-omeragaakbas
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test-analysis
Source: https://github.com/omeragaakbas/zoyare/tree/main/.claude/skills/ab-test-analysis
Command: npx skills add https://github.com/omeragaakbas/zoyare --skill ab-test-analysis-omeragaakbas

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps teams evaluate A/B and split-test results with statistical rigor so they can decide whether to ship, extend, or stop a variant without guessing.

Core Features & Use Cases

  • Statistical validation: Compute conversion rates, relative lift, p-values (two-tailed z-test / chi-squared), and 95% confidence intervals for control and variant groups.
  • Power & sample checks: Validate sample size against expected MDE using the n = (Z²α/2 × 2 × p × (1-p)) / MDE² formula and flag underpowered tests.
  • Operational guidance: Detect sample ratio mismatch, novelty/primacy effects, check guardrail metrics, and translate results into ship/extend/stop/investigate recommendations.
  • Reproducible analysis: Generate Python scripts to run calculations when raw CSV or analytics exports are provided and produce a concise results summary for stakeholders.

Quick Start

Analyze the attached experiment CSV to compute conversion rates, p-values, confidence intervals, and receive a ship/extend/stop recommendation.

Frequently Asked Questions about ab-test-analysis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I analyze A/B test results for statistical significance?▼

To analyze A/B test results for statistical significance, you compute conversion rates, relative lift, p-values using two-tailed z-tests or chi-squared tests, and 95% confidence intervals for control and variant groups.

How do I calculate sample size and power for conversion experiments?▼

You calculate sample size for conversion experiments using the n = (Z²α/2 × 2 × p × (1-p)) / MDE² formula to validate expected minimum detectable effects and flag underpowered tests.

Can I use CSV or Excel exports to compute split test p-values?▼

Yes, you can analyze raw CSV, Excel, or analytics exports to compute split test p-values, confidence intervals, and generate reproducible Python scripts for stakeholder reporting.

What is the best way to translate split test metrics into ship or stop recommendations?▼

Translating split test metrics into recommendations involves checking guardrail metrics, detecting sample ratio mismatch and novelty effects, then deriving ship, extend, stop, or investigate decisions.

Why does my A/B test show sample ratio mismatch and how do I detect it?▼

Sample ratio mismatch occurs when traffic allocation between control and variant groups deviates from expected proportions; operational guidance detects this alongside novelty and primacy effects during analysis.

Related Skills

Explore More Data & Analytics Skills→
OliverOuyangOliverOuyang
1

rta-exclude-strategy

Generate RTA exclusion strategies for V8 and V9RN cross-model datasets.

Community
Advanced
tarunccettarunccet
5

ab-test-analysis

Analyze A/B test results and generate ship, extend, or stop recommendations.

Community
Advanced
liangdabiaoliangdabiao
264

ab-testing-analyzer

Analyze AB test data to compute conversion rates and statistical significance.

Community
Advanced
olavocarvalhoolavocarvalho
1

ab-test

Analyze A/B test data with Frequentist and Bayesian significance testing.

Community
Advanced