skills.rest
Yvesdefaria
YvesdefariaCOMMUNITY
Explore all skills by this author →
GitHub

ab-testing

Designs and analyzes A/B tests and growth experimentation programs with statistical rigor.

Updated Aug 3, 2026
Search Tags:#growth#conversion optimization#hypothesis#experimentation#sample size#ab testing#statistical significance
One-click install
npx skills add https://github.com/Yvesdefaria/GymLab --skill ab-testing-yvesdefaria
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-testing
Source: https://github.com/Yvesdefaria/GymLab/tree/main/.opencode/skills/marketingskills/skills/ab-testing
Command: npx skills add https://github.com/Yvesdefaria/GymLab --skill ab-testing-yvesdefaria

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Teams often run experiments without clear hypotheses, stop tests too early, or misread results, leading to false conclusions and wasted traffic. This Skill provides a structured methodology for planning, running, and analyzing A/B tests so decisions are based on statistically valid evidence. ## Core Features & Use Cases - Hypothesis & Test Design: Builds structured hypotheses, selects test types (A/B, A/B/n, MVT, split URL), and defines primary, secondary, and guardrail metrics. - Sample Size & Duration Planning: Provides quick-reference sample size tables, duration formulas, and guidance on the peeking problem and sequential testing. - Growth Experimentation Program: Supports ICE prioritization, experiment velocity tracking, and a reusable experiment playbook for compounding wins. - Use Case: A team wants to test a new pricing page headline. The Skill calculates the required sample size from their 3% baseline conversion rate, defines metrics, warns against stopping early, and produces a documented test plan. ## Quick Start Ask the assistant to help design an A/B test for a specific page or change, providing your current conversion rate and traffic volume.

Frequently Asked Questions about ab-testing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate sample size for an A/B test?▼

Sample size depends on your baseline conversion rate, minimum detectable effect, significance level (usually 95%), and power (usually 80%). Use the quick reference tables in the sample size guide or calculators like Evan Miller's to get the required visitors per variant.

How long should I run an A/B test?▼

Run until you reach the pre-calculated sample size, with a minimum of one full week to capture day-of-week variation. Divide total required sample by daily traffic to estimate duration, and avoid tests longer than 4-8 weeks.

Can I stop an A/B test early when results look significant?▼

Stopping early after peeking at results inflates false positive rates and leads to wrong decisions. Pre-commit to your sample size and duration; if you need early-stopping flexibility, use sequential testing methods supported by tools like Optimizely or VWO.

What is the difference between A/B testing and multivariate testing?▼

A/B testing compares two versions with a single change, while multivariate testing (MVT) tests combinations of multiple elements simultaneously. MVT requires dramatically more traffic since combinations multiply, so sequential A/B tests are often better for lower-traffic sites.

What metrics should I track in an A/B test?▼

Define three tiers: a primary metric tied to your hypothesis that determines the winner, secondary metrics that explain why the change worked, and guardrail metrics that ensure nothing important gets worse during the test.

What should I do when my A/B test result is inconclusive?▼

An inconclusive result usually means the effect was smaller than your minimum detectable effect or the change was too subtle. Consider running longer for more sample, testing a bolder variant, or analyzing segments for hidden differences.

Related Skills

Explore More Marketing & Sales Skills→
KhaledaunKhaledaun

google-analytics

Analyzes Google Analytics metrics like sessions, users, bounce rates, and conversion rates.

Community
Intermediate
PrestonigoPrestonigo

ab-test-setup

Plan and execute A/B and multivariate experiments with sample size guidance.

Community
Advanced
vladosvlados
3

copywriter

Generate conversion-focused copy variants for sales letters, landing pages, emails, ads, and VSLs.

Community
Advanced
MonumentalSystemsMonumentalSystems
17

user-research

Convert user research activities into validated insights for product decisions.

Official
Intermediate