skills.rest
Productfculty-aipm
Productfculty-aipmCOMMUNITY
Explore all skills by this author →
GitHub

ab-test-design

Design A/B tests with sample size, duration, and analysis plans.

70|34|Updated Apr 7, 2026
Search Tags:#hypothesis#ab-testing#sample-size#experiment-design#statistical-significance#mde#analysis-plan
One-click install
npx skills add https://github.com/Productfculty-aipm/PM-Copilot-by-Product-Faculty --skill ab-test-design-productfculty-aipm
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test-design
Source: https://github.com/Productfculty-aipm/PM-Copilot-by-Product-Faculty/tree/main/skills/ab-test-design
Command: npx skills add https://github.com/Productfculty-aipm/PM-Copilot-by-Product-Faculty --skill ab-test-design-productfculty-aipm

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Helps product teams design rigorous A/B tests that produce trustworthy, actionable results rather than ambiguous or misleading outcomes by guiding hypothesis definition, metric selection, sample sizing, validity checks, and analysis planning.

Core Features & Use Cases

  • Falsifiable Hypotheses: Templates and guidance to craft a clear, mechanism-driven hypothesis that can be proven wrong.
  • Metric Selection & Guardrails: Choose a primary metric, define guardrail metrics, and ensure sensitivity and measurability within the test window.
  • Sample Size & Duration: Calculate required sample size from baseline conversion, MDE, power, and significance and estimate test duration; flag tests likely to run too long.
  • Validity & Analysis Plan: Pre-launch checks (SRM, assignment consistency, leakage), novelty effect considerations, and a pre-specified analysis plan including decision rules for inconclusive or marginal results.
  • Context-aware guidance: Reads product stage and analytics baseline to warn against testing pre-PMF products and to pull baseline conversion rates for calculations.

Quick Start

Design an A/B test to evaluate whether the new onboarding flow increases signup conversion by specifying the falsifiable hypothesis, primary metric, baseline conversion rate, minimum detectable effect, statistical power, and expected test duration.

Frequently Asked Questions about ab-test-design

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate sample size for an A/B test using baseline conversion and minimum detectable effect?▼

A/B test sample size is calculated by inputting baseline conversion rate, minimum detectable effect, statistical power, and significance level to estimate required traffic and test duration accurately.

How do I write a falsifiable hypothesis for a product experiment?▼

Writing a falsifiable hypothesis for a product experiment involves crafting a clear, mechanism-driven statement that predicts the causal impact of product changes on key metrics and can be proven wrong.

What guardrail metrics and validity checks do I need before launching an A/B test?▼

Before launching an A/B test, you need guardrail metrics to protect against negative impacts, plus validity checks like sample ratio mismatch (SRM), assignment consistency, and leakage detection.

Can I run an A/B test for a pre-PMF product or with low traffic?▼

Running an A/B test for a pre-PMF product or low-traffic scenarios is not recommended. The tool reads product stage to warn against testing and flags experiments likely to run too long.

How do I handle novelty effects and inconclusive results in experiment analysis?▼

Handle novelty effects and inconclusive A/B test results by applying pre-specified analysis plans and decision rules, ensuring you account for novelty effect considerations when evaluating marginal outcomes.

Related Skills

Explore More Product & Management Skills→
tuanqt98tuanqt98

19-ab-test-setup

Define hypotheses, compute sample sizes, and analyze statistical significance for A/B tests on campaigns with measurable metrics.

Community
Intermediate
SpaceZephyrSpaceZephyr
14

SPACE-experiment-designer

Generate end-to-end A/B experiment plans with sample size calculations and guardrail metrics.

Community
Advanced
jonathanmalkinjonathanmalkin
61

growth-audit

Analyze web app funnel metrics, identify bottlenecks, and design RICE-scored experiments.

Community
Advanced
sunnypatneedisunnypatneedi
11

product-market-fit

Measure and diagnose product-market fit using retention cohorts, Sean Ellis testing, and qualitative signals.

Community
Advanced