skills.rest
orqesa
orqesaOFFICIAL
Explore all skills by this author →
GitHub

experiment_design

Design product experiments with A/B tests, fake-door tests, and concierge MVPs.

Updated Aug 27, 2026
Search Tags:#experiment design#product validation#hypothesis testing#a/b testing#data-driven decisions#ai experimentation
One-click install
npx skills add https://github.com/orqesa/orqesa-roles --skill experiment-design-orqesa
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: experiment_design
Source: https://github.com/orqesa/orqesa-roles/tree/main/roles/cpo/skills/experiment-design
Command: npx skills add https://github.com/orqesa/orqesa-roles --skill experiment-design-orqesa

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps product teams design and execute rigorous experiments to validate product decisions, reducing the risk of costly mistakes without hindering development speed.

Core Features & Use Cases

  • Hypothesis Formulation: Guides users to create falsifiable hypotheses in a structured format.
  • Experiment Type Selection: Recommends appropriate methods like A/B tests, fake-door tests, and concierge MVPs based on learning needs and resources.
  • Metric Definition: Ensures clear definition of primary, secondary, and guardrail metrics.
  • Sample Size & Runtime Calculation: Provides guidance on statistical planning for A/B tests.
  • AI Feature Adaptation: Offers specific strategies for testing non-deterministic AI features, including eval-driven testing and qualitative assessment.

Quick Start

Use the experiment_design skill to create a hypothesis for testing a new onboarding flow.

Frequently Asked Questions about experiment_design

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design an A/B test for product validation?▼

A/B test design for product validation requires formulating a falsifiable hypothesis, defining primary and guardrail metrics, and calculating sample size to generate reliable signal for decisions.

What is the best way to test AI features that have non-deterministic outputs?▼

Testing non-deterministic AI features requires eval-driven testing and qualitative assessment strategies to generate reliable signal, adapting standard experiment design to handle variable AI outputs.

How do I calculate sample size and runtime for an A/B test?▼

Sample size and runtime calculation for an A/B test involves statistical planning based on your primary metrics and desired confidence level to ensure your experiment generates reliable signal.

When should I use a fake-door test versus a concierge MVP?▼

Use a fake-door test or concierge MVP based on your learning needs and resources; both methods validate product hypotheses early without full development, reducing the risk of costly mistakes.

What metrics do I need to define before running a product experiment?▼

Defining metrics for a product experiment requires establishing clear primary, secondary, and guardrail metrics to measure outcomes and protect against unintended negative impacts during testing.

How do I write a falsifiable hypothesis for product testing?▼

Writing a falsifiable hypothesis for product testing involves structuring your prediction in a specific format that clearly defines the expected change, the affected metric, and the measurable outcome.

Related Skills

Explore More Product & Management Skills→
nnplingggnnplinggg

epic-hypothesis

Frame testable if/then hypotheses with experiment plans and success metrics.

Community
Intermediate
liangdabiaoliangdabiao
264

ab-testing-analyzer

Analyze AB test data to compute conversion rates and statistical significance.

Community
Advanced
baasith6baasith6

adding-analytics

Integrate PostHog analytics into web applications for event tracking and feature flagging.

Community
Intermediate
bomberoxenviosdosruedasbomberoxenviosdosruedas

agency-ux-researcher

Conduct usability testing and user behavior analysis for design insights.

Community
Advanced