skills.rest
JDMAXilius
JDMAXiliusCOMMUNITY
Explore all skills by this author →
GitHub

ab-test-setup

Calculate sample sizes and analyze conversion rate A/B test results.

Updated Jun 27, 2026
Search Tags:#ab-testing#conversion-rate-optimization#experiment-design#statistical-significance#results-analysis#sample-size-calculation#test-plan-generation
One-click install
npx skills add https://github.com/JDMAXilius/Brand-building --skill ab-test-setup-jdmaxilius
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test-setup
Source: https://github.com/JDMAXilius/Brand-building/tree/main/skills/ab-test-setup
Command: npx skills add https://github.com/JDMAXilius/Brand-building --skill ab-test-setup-jdmaxilius

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Manually calculating A/B test sample sizes, designing rigorous test plans, and analyzing results with proper statistical validation is time-consuming and error-prone for growth and marketing teams, leading to underpowered tests or incorrect business decisions.

Core Features & Use Cases

  • Sample Size Calculation: Compute required sample sizes for conversion rate experiments with configurable significance levels, statistical power, and minimum detectable effect.
  • Test Plan Generation: Create comprehensive test plans with hypothesis documentation, timeline estimation, traffic allocation, and pre-launch checklists.
  • Statistical Result Analysis: Analyze test results with two-proportion z-tests, confidence intervals, effect size metrics, and clear ship/no-ship recommendations.
  • Use Case: A product team testing a new homepage CTA button can use this skill to calculate the required sample size, generate a full aligned test plan, and analyze final results to make a data-driven launch decision.

Quick Start

Use the ab-test-setup skill to calculate the required sample size for a homepage CTA A/B test with a 5% baseline conversion rate and 10% minimum detectable effect, then generate a full test plan and analyze the final results for a ship decision.

Frequently Asked Questions about ab-test-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate sample size for an A/B test with a specific minimum detectable effect?▼

A/B test sample size calculation requires your baseline conversion rate, minimum detectable effect, statistical significance level, and statistical power. The skill computes the required traffic volume to ensure your conversion rate experiment is properly powered.

How do I analyze A/B test results for statistical significance and conversion rate optimization?▼

Analyzing A/B test results for statistical significance involves applying two-proportion z-tests to your conversion data. The skill calculates confidence intervals, effect size metrics, and delivers actionable ship/no-ship recommendations for your variants.

Can I generate a complete A/B test plan including traffic allocation and timelines?▼

You can generate a complete A/B test plan that includes hypothesis documentation, timeline estimation, and traffic allocation. The skill outputs a structured pre-launch checklist to align growth teams before running controlled conversion experiments.

What statistical methods are used for conversion rate experiment analysis?▼

Conversion rate experiment analysis uses two-proportion z-tests to compare variant performance. The skill evaluates statistical significance by calculating confidence intervals and effect size metrics to validate your UI and pricing test decisions.

Why does my A/B test result in an underpowered experiment?▼

An A/B test becomes underpowered when the sample size is too small relative to the minimum detectable effect. The skill prevents this by calculating the exact required sample size needed upfront based on your desired statistical power and significance level.

Related Skills

Explore More Marketing & Sales Skills→
Gull-StackGull-Stack

ab-test-setup

Designs, plans, and analyzes A/B tests including hypothesis, metrics, and sample size calculations.

Official
Advanced
officebeatsofficebeats
8

ab-test-analysis

Analyze A/B test results for statistical significance and sample size validation.

Community
Advanced
systemifyautomationsystemifyautomation

ab-test-setup

Plan and design A/B tests with hypothesis generation and sample size calculation.

Community
Intermediate
hxk622hxk622
8

aeon

Perform time series classification, regression, clustering, forecasting, and anomaly detection with aeon.

Community
Advanced