skills.rest
azeemdigital
azeemdigitalCOMMUNITY
Explore all skills by this author →
GitHub

ab-test-setup

Design statistically valid A/B tests with sample size and duration planning.

Updated Apr 4, 2026
Search Tags:#conversion optimization#hypothesis#experimentation#sample size#ab testing#statistical significance#multivariate testing
One-click install
npx skills add https://github.com/azeemdigital/Claude-skills --skill ab-test-setup-azeemdigital
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test-setup
Source: https://github.com/azeemdigital/Claude-skills/tree/main/skills/ab-test-setup
Command: npx skills add https://github.com/azeemdigital/Claude-skills --skill ab-test-setup-azeemdigital

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you design A/B tests and experiments that produce trustworthy results instead of misleading guesses. It is useful when you need to compare versions of a page, feature, or message and want clear guidance on hypotheses, metrics, sample size, duration, and interpretation.

Core Features & Use Cases

  • Experiment design: Defines test type, hypothesis, variants, traffic allocation, and implementation approach for A/B, A/B/n, split URL, and multivariate tests.
  • Measurement planning: Guides selection of primary, secondary, and guardrail metrics so tests are tied to business outcomes and protected against harmful side effects.
  • Statistical rigor: Helps estimate sample size, test duration, and significance thresholds while warning against early stopping and other common testing mistakes.
  • Use cases: Plan a homepage headline test, evaluate whether a pricing page experiment has enough traffic, structure a signup form test around quantity versus lead quality, or decide whether a multivariate test is feasible.

Quick Start

Ask the ab-test-setup skill to design an experiment for the page or feature you want to test, including the hypothesis, metrics, sample size, duration, and decision criteria.

Frequently Asked Questions about ab-test-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate the right sample size and duration for an A/B test?▼

To plan an A/B test, define your hypothesis, select primary and guardrail metrics, estimate sample size based on significance thresholds, set test duration, and establish clear decision criteria for interpreting results.

What is the difference between A/B/n, split URL, and multivariate testing?▼

A/B/n compares multiple variants against a control, split URL tests redirect traffic to different pages entirely, and multivariate testing evaluates combinations of page elements to identify specific interaction effects.

How do I select primary, secondary, and guardrail metrics for conversion optimization experiments?▼

Primary metrics tie directly to your main business outcome, secondary metrics provide contextual behavioral insights, and guardrail metrics protect against harmful side effects like increased load time or decreased retention.

Why does early stopping ruin statistical significance in A/B testing?▼

Early stopping during A/B testing introduces bias by ignoring pre-calculated test duration requirements, inflating false positive rates, and compromising the anti-peeking discipline necessary for trustworthy experiment evaluation.

Does my pricing page have enough traffic to run a statistically valid experiment?▼

Evaluating whether a page has enough traffic for a valid experiment involves estimating required sample size against current traffic volume, expected effect size, and desired statistical significance thresholds.

Related Skills

Explore More Marketing & Sales Skills→
sicktasticsicktastic

ab-test-setup

Plan statistically sound A/B tests with sample size and duration calculations.

Community
Advanced
omeragaakbasomeragaakbas

ab-test-analysis

Analyze A/B test results to compute statistical significance and practical impact.

Community
Intermediate
tomtoto757tomtoto757
14

ab-testing-ecommerce

Run controlled A/B experiments with sample size calculation and variant assignment.

Community
Intermediate
zocomputerzocomputer
45

ab-test-setup

Plan, design, and implement A/B tests with hypothesis formulation and sample size calculation.

Official
Advanced