skills.rest
samChang72
samChang72COMMUNITY
Explore all skills by this author →
GitHub

ab-test-setup

Design and conduct A/B tests with hypothesis formation and sample size calculation.

1|Updated Feb 2, 2026
Search Tags:#A/B testing#test design#experimentation#statistical analysis#hypothesis development
One-click install
npx skills add https://github.com/samChang72/custom-skills --skill ab-test-setup-samchang72
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test-setup
Source: https://github.com/samChang72/custom-skills/tree/main/gemini/skills/ab-test-setup
Command: npx skills add https://github.com/samChang72/custom-skills --skill ab-test-setup-samchang72

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires statistical, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill assists in the planning, design, and implementation of A/B tests and experiments, ensuring statistical validity and actionable results.

Core Features & Use Cases

  • Hypothesis Framework: A structured approach to develop strong hypotheses for A/B testing.
  • Test Types: Guidance on choosing the right test type (A/B, A/B/n, MVT) based on the specific goals.
  • Sample Size Calculation: Tools and guidance for determining the optimal sample size for the test.
  • Metrics Selection: Best practices for choosing primary, secondary, and guardrail metrics.
  • Variant Design: Best practices for designing clear and effective variants.
  • Traffic Allocation: Strategies for allocating traffic to different variants.
  • Implementation Approaches: Options for client-side and server-side testing with feature flags.
  • Running the Test: A step-by-step guide on how to execute the test, including a pre-launch checklist.
  • Analyzing Results: How to interpret statistical significance, practical significance, and analyze results for actionable insights.

Quick Start

Design an A/B test for improving user engagement on a webpage by adjusting the call-to-action button color.

Frequently Asked Questions about ab-test-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design an A/B test with statistical rigor?▼

To design an A/B test with statistical rigor, use a structured framework to form hypotheses, select appropriate test types, calculate optimal sample sizes, and choose primary, secondary, and guardrail metrics.

What is the best way to calculate sample size for A/B testing?▼

The best way to calculate sample size for A/B testing involves using statistical tools to determine the optimal number of participants needed per variant to ensure your experiment results are statistically valid.

How do I choose between A/B, A/B/n, and MVT test types?▼

Choosing between A/B, A/B/n, and MVT test types depends on your specific goals: standard A/B tests compare two variants, A/B/n compares multiple, and MVT evaluates multiple element changes simultaneously.

Do I need statistical knowledge for A/B test significance testing?▼

Yes, you need statistical knowledge for A/B test significance testing. Interpreting statistical significance, practical significance, and analyzing results for actionable insights requires a solid understanding of statistics.

How do I implement traffic allocation and feature flags for experiments?▼

Implement traffic allocation by applying strategies to distribute visitors across variants, and use feature flags for client-side or server-side testing to control which users see which experiment variant.

What metrics should I track when running an A/B test?▼

When running an A/B test, track primary metrics to measure main goals, secondary metrics for additional insights, and guardrail metrics to ensure the experiment does not negatively impact existing systems.

Related Skills

Explore More Software Engineering Skills→
viniruggeriviniruggeri

hypothesis-generation

Generate structured, testable hypotheses with predictions and experimental designs.

Community
Advanced
w95w95
145

statistical-analysis

Apply descriptive statistics, trend analysis, outlier detection, and hypothesis testing to datasets.

Community
Intermediate
matheus3301matheus3301

ab-test-setup

Plan, design, and implement A/B tests for conversion rate optimization.

Community
Intermediate
sarahxu0205sarahxu0205
11

ab-testing-design

Design A/B tests with sample size calculations and statistical significance interpretation.

Community
Intermediate