ab-test-calculator

Calculate statistical significance for A/B tests using Chi-square, Z-test, or T-test.

86|18|Updated Dec 14, 2025

Search Tags:#conversion rate #a/b testing #sample size #power analysis #statistical significance #bayesian analysis

Ready to run

npx skills add https://github.com/dkyazzentwatwa/chatgpt-skills --skill ab-test-calculator

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: ab-test-calculator
Source: https://github.com/dkyazzentwatwa/chatgpt-skills/tree/main/ab-test-calculator
Command: npx skills add https://github.com/dkyazzentwatwa/chatgpt-skills --skill ab-test-calculator

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires scipy, numpy, and includes scripts (resource) components.

What problem does it solve?

This Skill helps you determine if the results of your A/B tests are statistically significant, ensuring you make data-driven decisions with confidence.

Core Features & Use Cases

Statistical Significance Testing: Use Chi-square, Z-test, or T-test to compare conversion rates between variants.
Sample Size & Power Analysis: Calculate the necessary sample size for your test or determine the power of an existing test.
Bayesian Analysis: Understand the probability of one variant being better than another.
Use Case: After running an A/B test on your website's call-to-action button, use this Skill to confirm if the observed increase in clicks for the new button is statistically significant or just due to random chance.

Quick Start

Calculate the statistical significance for an A/B test with 10000 control visitors and 500 conversions, and 10000 variant visitors with 550 conversions.

Frequently Asked Questions about ab-test-calculator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I calculate statistical significance for my A/B test conversion rates?▼

Calculate statistical significance for A/B tests by comparing variant conversion rates using Chi-square, Z-test, or T-test. Input visitor and conversion counts for both groups to determine if observed differences are meaningful or due to random chance.

How do I estimate the sample size needed for an A/B test?▼

Estimate required sample size for an A/B test using power analysis features. Provide expected baseline conversion rate and minimum detectable effect to calculate the visitor count needed for statistically valid results.

Can I use Bayesian analysis to find the probability that one variant is better than another?▼

Bayesian analysis calculates the probability that one variant outperforms another. This method evaluates underlying conversion rate distributions to give a direct probability percentage for decision-making.

Does this A/B test calculator support experiments with multiple variants?▼

The calculator supports multiple variants in A/B testing. It compares several groups simultaneously using the same statistical testing frameworks to evaluate conversion rate differences across all tested variations.

What statistical libraries are required to compute confidence intervals and power analysis?▼

Computing confidence intervals and power analysis requires scipy, numpy, and statsmodels. These libraries handle the underlying statistical distributions and accurate computations for conversion rate intervals.

When should I use a Chi-square test versus a T-test for A/B testing?▼

Use Chi-square tests for categorical conversion data comparing frequencies between variants. T-tests are applied when comparing the means of continuous metrics across test groups to determine statistical significance.