ab-testing-statistics

Select statistical tests and plan A/B experiments with power analysis.

Updated Jan 15, 2026

Search Tags:#statistics #ab-testing #hypothesis-testing #effect-size #p-value #power-analysis #a/b-test

Ready to run

npx skills add https://github.com/KaiserWhoLearns/skillsbench --skill ab-testing-statistics

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: ab-testing-statistics
Source: https://github.com/KaiserWhoLearns/skillsbench/tree/main/tasks/ab-testing-stats/environment/skills/ab-testing-statistics
Command: npx skills add https://github.com/KaiserWhoLearns/skillsbench --skill ab-testing-statistics

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

An experienced data analyst needs a consistent, statistically valid method for interpreting A/B test results, ensuring correct test selection and accurate conclusions.

Core Features & Use Cases

Select appropriate tests for binary metrics (two-proportion z-test) and continuous metrics (Welch's t-test).
Perform multiple comparison corrections and calculate effect sizes and relative lift.
Plan experiments with power analysis to determine required sample sizes and guide reporting.

Quick Start

Run the recommended A/B analysis on your experiment data and interpret the results.

Frequently Asked Questions about ab-testing-statistics

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I select the right statistical test for my A/B test results?▼

Selecting the right statistical test for A/B testing depends on your metric type: use a two-proportion z-test for binary metrics and Welch's t-test for continuous metrics to ensure valid conclusions.

What is power analysis and how does it apply to A/B testing?▼

Power analysis in A/B testing is a technique used during experiment planning to determine the required sample size, ensuring your test has enough statistical power to detect a meaningful effect size before you begin data collection.

How do I handle multiple comparisons and calculate effect size in A/B tests?▼

To handle multiple comparisons in A/B testing, apply multiple comparison corrections to control error rates, while calculating effect sizes and relative lift provides a standardized measure of the actual magnitude of the observed difference.

What is the best way to interpret p-values and report A/B test findings?▼

The best way to interpret p-values in A/B testing is to follow a reproducible workflow that guides data preparation, test selection, significance interpretation, and reporting, ensuring your conclusions are statistically sound.

Do I need a large dataset to run a valid A/B test analysis?▼

You need a dataset large enough to meet the sample size requirements determined by your power analysis; running A/B tests without calculating required sample sizes risks false negatives and invalid conclusions.