ab-test-setup

Plan A/B tests with hypotheses, sample size calculations, and variant design.

Updated Jan 24, 2026

Search Tags:#analysis #statistics #hypothesis #a/b testing #sample-size #experimental-design #ab-test

Ready to run

npx skills add https://github.com/seflless/dot-agents --skill ab-test-setup-seflless

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: ab-test-setup
Source: https://github.com/seflless/dot-agents/tree/main/skills/ab-test-setup
Command: npx skills add https://github.com/seflless/dot-agents --skill ab-test-setup-seflless

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps teams plan, design, and execute rigorous A/B tests to validate product changes with statistical rigor.

Core Features & Use Cases

Framework for defining hypotheses, selecting test types (A/B, A/B/n, MVT), and calculating required sample sizes.
Guidance on designing variants, traffic allocation, and interpreting results across pages, features, and funnels.
Use Case: when evaluating pricing changes, feature toggles, or UI copy, this Skill provides a repeatable method to measure impact.

Quick Start

Draft a test hypothesis, choose the primary metric, and run the sample-size calculation using the references in this skill.

Frequently Asked Questions about ab-test-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I calculate the required sample size for an A/B test?▼

To calculate the required sample size for an A/B test, you need to define your primary metric, establish the expected effect size, and apply statistical formulas to ensure your test has enough power to detect meaningful changes.

What type of A/B test should I use for evaluating multiple UI variants?▼

For evaluating multiple UI variants, an A/B/n test or Multivariate Test (MVT) is appropriate. These test types allow you to compare several variants simultaneously to measure their distinct impact on user experience and primary metrics.

How do I design a statistically valid A/B test hypothesis?▼

A statistically valid A/B test hypothesis clearly states the expected change, the variant being tested, and the primary metric impacted. This structured framework ensures your experimental design accurately measures the effect of product changes.

Can I use this A/B testing framework for pricing and copy experiments?▼

Yes, you can use this A/B testing framework for pricing, copy, and feature toggle experiments. It provides a repeatable method to design variants, allocate traffic, and measure the statistical impact across various user experiences.

What's the best way to interpret A/B test results across different funnels?▼

The best way to interpret A/B test results across funnels is to analyze the statistical significance of your primary metric and evaluate the impact on user behavior. This ensures you validate product changes with statistical rigor.

When should I avoid running an A/B test?▼

You should avoid running an A/B test when you lack sufficient traffic to reach the calculated sample size or when the expected effect size is too small to achieve statistical significance within a practical timeframe.