skills.rest
Avihusitton
AvihusittonCOMMUNITY
Explore all skills by this author →
GitHub

ab-test-setup

Design, execute, and analyze statistically valid A/B tests with sample size calculation.

1|Updated May 10, 2026
Search Tags:#ab-testing#hypothesis-testing#growth-experiments#conversion-optimization#statistical-significance#ice-scoring#experimentation-program
One-click install
npx skills add https://github.com/Avihusitton/gil-therapy --skill ab-test-setup-avihusitton
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test-setup
Source: https://github.com/Avihusitton/gil-therapy/tree/main/.agents/skills/ab-test-setup
Command: npx skills add https://github.com/Avihusitton/gil-therapy --skill ab-test-setup-avihusitton

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill eliminates the risk of running inconclusive, statistically invalid A/B tests that waste team time and resources, by providing structured, best-practice guidance for every stage of experimentation from hypothesis creation to result analysis and program scaling.

Core Features & Use Cases

  • Structured Hypothesis Framework: Build testable, data-backed hypotheses using a proven template to avoid vague, low-impact test ideas.
  • Statistical Rigor Tools: Calculate required sample sizes, test durations, and significance thresholds to ensure results are reliable and actionable.
  • Full Experimentation Program Support: Guidance for building a continuous growth experimentation practice including ICE prioritization, experiment playbooks, and velocity tracking. Use case: A growth manager wanting to test a new signup flow can use this Skill to define success metrics, calculate how long the test needs to run, avoid the peeking problem, and document learnings for future tests.

Quick Start

Use the ab-test-setup skill to design a statistically valid A/B test for your homepage CTA, including a clear hypothesis, required sample size, and guardrail metrics to track.

Frequently Asked Questions about ab-test-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate the required sample size for an A/B test to ensure statistical significance?▼

To calculate the required sample size for an A/B test, you must define your significance threshold, baseline conversion rate, and minimum detectable effect to ensure results are reliable and statistically valid.

What is the best way to formulate a testable hypothesis for growth experiments?▼

The best way to formulate a testable hypothesis for growth experiments is using a structured framework template that builds data-backed hypotheses, avoiding vague test ideas and ensuring clear metric definition for conversion optimization.

How do I avoid the peeking problem when running A/B tests?▼

To avoid the peeking problem in A/B testing, you must calculate the required test duration and sample size beforehand and commit to running the experiment for that full period before evaluating statistical significance.

Can I use ICE scoring to prioritize my conversion rate optimization tests?▼

Yes, you can use ICE scoring to prioritize conversion rate optimization tests by evaluating the impact, confidence, and ease of each experiment idea to build a continuous, high-velocity growth experimentation program.

What guardrail metrics should I track during A/B test setup?▼

During A/B test setup, you should track guardrail metrics alongside your primary success metrics to prevent negative impacts on core user experience or business outcomes while running your growth experiments.

Related Skills

Explore More Marketing & Sales Skills→
yonatangrossyonatangross
217

product-analytics

Evaluate A/B tests, cohort retention, and funnel metrics for product decisions.

Community
Advanced
fianchettogiannifianchettogianni

ab-testing

Design statistically valid A/B tests with sample size and significance calculations.

Community
Advanced
TheWayWithinTheWayWithin
13

saas-analytics

Implements product analytics for SaaS apps via provider-agnostic interfaces.

Community
Intermediate
brainbytes-devbrainbytes-dev
3

ad-creative-testing

Plan A/B, multivariate, and champion/challenger tests for ad campaigns.

Community
Advanced