skills.rest
kalilurrahman
kalilurrahmanCOMMUNITY
Explore all skills by this author →
Website𝕏@krahmanGitHub

ab-test-design

Designs complete A/B test experiment plans with hypotheses, metrics, sample sizes, and success criteria.

1|Updated Mar 21, 2026
Search Tags:#product management#experiment design#conversion optimization#hypothesis#sample size#ab testing#guardrail metrics
One-click install
npx skills add https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts --skill ab-test-design-kalilurrahman
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test-design
Source: https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts/tree/main/09-product-management/ab-test-design
Command: npx skills add https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts --skill ab-test-design-kalilurrahman

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Turning a vague test idea into a rigorous, implementable experiment plan is hard — teams often skip sample size math, pick too many metrics, or run tests too long. This Skill converts a test idea into a complete experiment design that an engineer can build and an analyst can evaluate. ## Core Features & Use Cases - Structured Hypothesis Formulation: Generates hypotheses with predicted magnitude using the format "If we [change], then [metric] will [direction] by [amount] because [reason]". - Metric Framework: Defines one primary decision metric, 2-3 secondary metrics, and mandatory guardrail metrics that must not degrade. - Sample Size & Duration Estimation: Calculates required sample per variant from baseline rate, MDE, 95% significance, and 80% power, using Bash with Python, and flags tests exceeding 4 weeks. - Use Case: A product manager wants to test a new checkout button color. The Skill produces a full experiment document with hypothesis, metrics table, variant specs, sample size, duration, success criteria, risks, and a pre-launch checklist. ## Quick Start Ask the AI to design an A/B test for changing the signup flow from two steps to one, including sample size and success criteria.

Frequently Asked Questions about ab-test-design

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design an A/B test from a hypothesis?▼

Provide the change you want to test, and the Skill generates a full experiment design: a structured hypothesis with predicted magnitude, one primary metric, guardrail metrics, variant descriptions, sample size, duration, and success criteria.

How to calculate sample size for an A/B test?▼

Sample size is estimated from the baseline conversion rate, minimum detectable effect (MDE), 95% statistical significance, and 80% power. The Skill uses Bash with Python to compute the required sample per variant when baseline rate and MDE are known.

What are guardrail metrics in A/B testing?▼

Guardrail metrics are measurements that must not degrade during a test, such as page load time or revenue per user. Every experiment design includes 2-3 guardrails with explicit thresholds, and the test stops early if a guardrail drops beyond its limit.

How long should an A/B test run?▼

Duration is estimated by dividing required sample size by daily traffic. If the estimate exceeds 4 weeks, the Skill flags it as a validity risk and suggests narrowing the audience, increasing the MDE, or testing on a higher-traffic page.

When should you not run an A/B test?▼

Avoid tests that are too small to detect a meaningful effect, too broad to learn from, or testing changes that should simply be shipped. The Skill pushes back on weak test ideas and asks clarifying questions when the input is vague.

Related Skills

Explore More Product & Management Skills→
aitytechaitytech
595

ab-test-setup

Designs A/B tests with hypotheses, sample size calculations, and statistical analysis plans.

Official
Intermediate
PostHogPostHog
713

creating-experiments

Creates draft A/B test experiments in PostHog with feature flags, rollout splits, and metrics.

Official
Intermediate
PostHogPostHog
713

scanning-experiments-with-replay-vision

Creates Replay Vision scanners scoped to an experiment's exposed sessions for per-variant behavioral analysis.

Official
Advanced
delta-and-betadelta-and-beta

tester-experiments

Run Braze A/B and feature flag experiments using random_bucket_number segments.

Community
Advanced