skills.rest
Pinkycherry
PinkycherryCOMMUNITY
Explore all skills by this author →
GitHub

ab-testing

Designs and analyzes A/B tests and growth experimentation programs with statistical rigor.

Updated Aug 2, 2026
Search Tags:#growth#conversion optimization#hypothesis#experimentation#sample size#ab testing#statistical significance
One-click install
npx skills add https://github.com/Pinkycherry/newbusinessideas3 --skill ab-testing-pinkycherry
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-testing
Source: https://github.com/Pinkycherry/newbusinessideas3/tree/main/.claude/skills/ab-testing
Command: npx skills add https://github.com/Pinkycherry/newbusinessideas3 --skill ab-testing-pinkycherry

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Teams often run experiments without clear hypotheses, stop tests too early, or misread results, leading to false conclusions and wasted traffic. This Skill provides a structured framework for designing statistically valid A/B tests, calculating sample sizes, and building a continuous experimentation program. ## Core Features & Use Cases - Hypothesis-Driven Test Design: Structures every test around a formal hypothesis with defined primary, secondary, and guardrail metrics. - Sample Size & Duration Planning: Provides quick-reference tables and duration formulas so tests reach statistical significance before being called. - Growth Experimentation Program: Covers ICE prioritization, experiment velocity tracking, and a playbook format for compounding learnings across tests. - Use Case: A product team wants to test a new pricing page headline. The Skill calculates the required sample size from their baseline conversion rate, defines guardrail metrics like refund rate, warns against peeking at early results, and produces a documented test plan. ## Quick Start Ask the assistant to help design an A/B test for a specific page or feature, providing your current conversion rate and traffic volume.

Frequently Asked Questions about ab-testing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate sample size for an A/B test?▼

Sample size depends on your baseline conversion rate, minimum detectable effect, significance level (usually 95%), and power (usually 80%). Use the quick reference tables in the sample size guide or online calculators like Evan Miller's to find the required visitors per variant.

How long should I run an A/B test?▼

Run tests until you reach the pre-calculated sample size, with a minimum of one full week to capture day-of-week variation. Divide total required sample by daily traffic to estimate duration, and avoid tests longer than 4-8 weeks.

Can I stop an A/B test early when one variant is winning?▼

Stopping early based on peeked results inflates false positive rates and leads to wrong decisions. Commit to the pre-calculated sample size, or use sequential testing methods supported by tools like Optimizely or PostHog if early stopping is required.

What is the difference between A/B testing and multivariate testing?▼

A/B tests compare two versions with a single change, while multivariate tests evaluate combinations of multiple elements simultaneously. Multivariate tests require dramatically more traffic since combinations multiply, so sequential A/B tests are often better for lower-traffic sites.

What metrics should I track in an A/B test?▼

Define three tiers: a primary metric tied directly to your hypothesis, secondary metrics that explain why the change worked, and guardrail metrics that catch negative side effects like increased refunds or support tickets.

What should I do when my site lacks traffic for A/B testing?▼

Increase your minimum detectable effect to require less sample, reduce the number of variants, test higher-traffic pages earlier in the funnel, or rely on qualitative research instead of running an underpowered test.

Related Skills

Explore More Marketing & Sales Skills→
KhaledaunKhaledaun

google-analytics

Analyzes Google Analytics metrics like sessions, users, bounce rates, and conversion rates.

Community
Intermediate
PrestonigoPrestonigo

ab-test-setup

Plan and execute A/B and multivariate experiments with sample size guidance.

Community
Advanced
omeragaakbasomeragaakbas

ab-test-analysis

Analyze A/B test results to compute statistical significance and practical impact.

Community
Intermediate
MonumentalSystemsMonumentalSystems
17

user-research

Convert user research activities into validated insights for product decisions.

Official
Intermediate