skills.rest
pisithrps
pisithrpsCOMMUNITY
Explore all skills by this author →
𝕏@pisithrpsGitHub

experiment-decision

Decide between A/B testing and shipping based on reversibility, impact, and risk.

Updated Mar 11, 2026
Search Tags:#risk-assessment#experimentation#ab-testing#hypothesis-driven#product-decision#power-calculation
One-click install
npx skills add https://github.com/pisithrps/yapzee --skill experiment-decision-pisithrps
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: experiment-decision
Source: https://github.com/pisithrps/yapzee/tree/main/.claude/skills/experiment-decision
Command: npx skills add https://github.com/pisithrps/yapzee --skill experiment-decision-pisithrps

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Helps product teams decide whether to run an A/B test, ship with monitoring, or simply release a change by evaluating reversibility, hypothesis quality, detectable impact, and risk versus cost. It prevents wasting engineering and analysis time on underpowered experiments and reduces slow decision cycles caused by testing everything.

Core Features & Use Cases

  • Decision tree guidance: Walks through reversibility, hypothesis presence, detectability (power), and risk to reach a clear recommendation.
  • Power and ROI assessment: Provides rules of thumb and sample-size guidance to determine if an experiment is practical given traffic and baseline rates.
  • Risk & cost tradeoffs: Compares engineering and opportunity costs of testing versus shipping and recommends monitoring and rollback plans.
  • Use cases: Pricing experiments, checkout changes, UI/messaging tweaks, feature launches, and optimization work where teams must weigh speed against evidence.

Quick Start

Invoke /experiment-decision and describe the feature, expected impact (metric and estimate), and your main concern about risk or reversibility.

Frequently Asked Questions about experiment-decision

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
When should I run an A/B test versus just shipping a product change?▼

Run an A/B test when the change is irreversible, high-risk, or has a clear hypothesis with measurable success metrics. Just ship the change if it is easily reversible, low-risk, or when baseline traffic is too low for adequate power calculation.

How do I know if my baseline traffic is high enough for A/B testing?▼

Assess baseline traffic by performing a power calculation. You need sufficient baseline conversion data and traffic volume to detect the expected impact. If sample size requirements are impractical for your traffic, testing is underpowered and shipping with monitoring is recommended.

What information do I need before running a product experiment?▼

Before running an experiment, define a clear hypothesis, identify measurable success metrics, gather baseline traffic and conversion data for power calculations, and assess reversibility and stakeholder risk to produce a valid recommendation.

How do I evaluate the risk and cost tradeoffs of experimentation?▼

Evaluate risk and cost tradeoffs by comparing engineering and opportunity costs of testing versus shipping directly. The decision tree weighs the cost of delayed shipping against the risk of deploying an untested change, recommending monitoring and rollback plans for high-risk reversals.

Does this approach work for pricing and onboarding optimization experiments?▼

Yes, this approach applies to pricing experiments, checkout changes, UI tweaks, messaging, onboarding, and optimization work. It helps weigh speed against evidence for any product decision where teams must balance reversibility and risk.

Why do my experiments keep producing inconclusive results?▼

Experiments produce inconclusive results when they are underpowered due to low baseline traffic or very small expected impact. If sample size requirements exceed practical limits, the decision framework recommends shipping with monitoring instead of wasting analysis time.

Related Skills

Explore More Product & Management Skills→
tuanqt98tuanqt98

19-ab-test-setup

Define hypotheses, compute sample sizes, and analyze statistical significance for A/B tests on campaigns with measurable metrics.

Community
Intermediate
jmmLjmmL

finishing-a-development-branch

Evaluate test results and present four git integration options for a feature branch.

Community
Intermediate
PiercingXXPiercingXX
1

ops-canary

Monitor post-deploy canary signals and trigger rollback on quality degradation.

Community
Intermediate
GuitarAlchemistGuitarAlchemist

ix-bandit

Simulate multi-armed bandit algorithms with epsilon-greedy, UCB1, and Thompson sampling.

Official
Intermediate