skills.rest
lofty-thoughts
lofty-thoughtsOFFICIAL
Explore all skills by this author →
WebsiteGitHub

ab-test-setup

Design statistically valid A/B tests with sample size and duration calculations.

Updated Feb 21, 2026
Search Tags:#hypothesis#ab-testing#sample-size#conversion-rate#experiment-design#metric-selection#variant-testing
One-click install
npx skills add https://github.com/lofty-thoughts/botdaddy --skill ab-test-setup-lofty-thoughts
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test-setup
Source: https://github.com/lofty-thoughts/botdaddy/tree/main/seed/skills/ab-test-setup
Command: npx skills add https://github.com/lofty-thoughts/botdaddy --skill ab-test-setup-lofty-thoughts

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Helps teams plan, design, and run A/B tests and experiments that produce statistically valid, actionable results by removing ambiguity around hypotheses, sample size, metrics, and analysis steps. Reduces common mistakes like peeking, underpowered tests, and poorly chosen primary metrics so decisions are data-driven and defensible.

Core Features & Use Cases

  • Hypothesis Framework: Structured template for writing clear, testable hypotheses tied to measurable outcomes.
  • Sample Size & Duration Guidance: Quick reference tables, duration formulas, and adjustments for multiple variants and low-traffic scenarios.
  • Design & Traffic Allocation: Recommendations for A/B, A/B/n, multivariate, and split-URL tests plus safe allocation strategies.
  • Metrics & Guardrails: Help selecting primary, secondary, and guardrail metrics and interpreting statistical significance and effect size.
  • Implementation Checklist & Templates: Pre-launch QA, tracking verification, documentation templates, and analysis/report templates for stakeholder communication.
  • Use Cases: Landing page CTA tests, pricing experiments, signup funnel changes, feature rollout experiments, and multivariate layout tests.

Quick Start

Describe the change, baseline conversion, traffic level, target minimum detectable effect, and primary metric and ask for a test plan with hypothesis, sample size estimate, variant descriptions, traffic split, and analysis checklist.

Frequently Asked Questions about ab-test-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate the right sample size and duration for an A/B test?▼

A/B test hypothesis frameworks use structured templates tying proposed changes to measurable outcomes. This Skill provides a clear format to write testable hypotheses, ensuring your product or marketing experiments are designed around specific, quantifiable metrics rather than ambiguous goals.

What is the best way to choose primary, secondary, and guardrail metrics for split URL tests?▼

Selecting split URL test metrics involves defining a primary conversion goal alongside secondary metrics and guardrails. This Skill helps you choose these metrics to interpret statistical significance and effect size while protecting against unintended negative impacts on your signup funnels or pricing flows.

Can I use this to plan A/B/n and multivariate experiments for low-traffic landing pages?▼

Yes, you can plan A/B/n and multivariate experiments for low-traffic landing pages using specialized traffic allocation strategies. The Skill adjusts sample size calculations and variant definitions to handle scenarios where baseline traffic volumes are a limiting factor for reaching significance.

Why does underpowered variant testing lead to inconclusive conversion rate optimization results?▼

Underpowered variant testing produces inconclusive conversion rate optimization results because the sample size is too small to detect the minimum effect size. This Skill prevents underpowered tests by calculating required traffic volumes and duration upfront, avoiding premature peeking at data.

What should be included in an A/B testing pre-launch checklist for feature flag rollouts?▼

An A/B testing pre-launch checklist for feature flag rollouts should include tracking verification, variant definitions, and traffic allocation strategies. This Skill provides a comprehensive QA checklist and documentation templates to ensure implementation and analysis steps are fully verified before launch.

Related Skills

Explore More Product & Management Skills→
tuanqt98tuanqt98

19-ab-test-setup

Define hypotheses, compute sample sizes, and analyze statistical significance for A/B tests on campaigns with measurable metrics.

Community
Intermediate
shalevaminshalevamin

ab-test-analysis

Analyze A/B test results for statistical significance and rollout recommendations.

Community
Advanced
executiveusaexecutiveusa
1

ab-test-setup

Plan A/B tests with sample sizes, traffic allocation, and implementation checklists.

Community
Intermediate
docaohieu2808docaohieu2808

ab-test-setup

Plan and implement A/B and multivariate tests with Python-based statistical analysis.

Community
Intermediate