ab-test-setup

Design statistically valid A/B, multivariate, and split test plans with hypotheses and metrics.

Updated Mar 17, 2026
One-click install
npx skills add https://github.com/TomWall1/WageCheck --skill ab-test-setup-tomwall1
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: ab-test-setup
Source: https://github.com/TomWall1/WageCheck/tree/main/.agents/skills/ab-test-setup
Command: npx skills add https://github.com/TomWall1/WageCheck --skill ab-test-setup-tomwall1

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Planning experimentation is messy and risky when teams skip context, fail to measure the right metrics, or stop tests early; this Skill compiles the right questions about hypotheses, context notes, traffic, and constraints so you only launch tests that answer meaningful questions.

Core Features & Use Cases

  • Hypothesis framework: Guides you to turn observations into experiments with clear predictions, expected audiences, and metric-based success criteria while encouraging you to review product marketing context before asking.
  • Sample sizing, traffic, and execution planning: Connects baseline rates with lifts, references the included sample-size guide, and reminds you to budget traffic splits, durations, peeking safeguards, and implementation approaches for client- or server-side setups.
  • Metrics and results discipline: Clarifies primary, secondary, and guardrail metrics, documents pre-launch checklists, and links to test templates so you can log learnings, interpret significance, and decide whether to ship.

Quick Start

Ask the skill to design a hypothesis and test plan for the change you want to evaluate on your page.

Frequently Asked Questions about ab-test-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design an A/B test plan with proper hypothesis and metrics?

To design an A/B test plan, document your hypothesis with clear predictions, expected audiences, and metric-based success criteria. Select primary, secondary, and guardrail metrics to measure impact accurately before launching any variations.

How do I calculate sample size and traffic allocation for split tests?

Calculate sample size for split tests by connecting baseline conversion rates with expected lift, then budget your traffic splits and test duration accordingly to ensure statistical significance without early peeking.

What are guardrail metrics and why do I need them for experiment design?

Guardrail metrics are safety indicators monitored during experiment design to prevent negative impacts on user experience or business health. You need them to catch unintended consequences while testing variations.

Can I use this approach for multivariate testing on pricing and feature funnels?

Yes, you can use this experimentation framework for multivariate testing on pricing and feature funnels. It targets product and marketing teams planning experiments across landing pages to compare multiple variations simultaneously.

What is the best way to stop peeking at A/B test results early?

The best way to prevent early peeking at A/B test results is to calculate the required sample size and duration upfront, then enforce pre-launch checklists and peeking safeguards before evaluating statistical significance.