feature-experiment-design

What problem does it solve?

It prevents teams from running A/B tests that cannot answer “did this work?” by enforcing pre-registration discipline, correct metric choice, and statistically valid decision criteria before the experiment starts.

Core Features & Use Cases

• Pre-registration & hypothesis planning: Defines the hypothesis, primary metric, direction, success criterion, sample size, and guardrail metrics so analysis can’t be retrofitted after results.
• Metric, MDE, and sample-size design: Chooses a primary metric that is sensitive, measurable at the test time horizon, causally connected to the business outcome, and calculable from collected data; computes MDE and estimated run time.
• Framework & integrity choices: Guides frequentist vs Bayesian decisions, warns against peeking/sequential misuse, and recommends variance reduction (e.g., CUPED) and heterogeneous treatment effect reporting with pre-registered segments.

Quick Start

Use feature-experiment-design to produce a pre-registered A/B test plan by asking: “Design an experiment behind a feature flag with a primary metric, guardrails, MDE, sample size, and a ship-or-no-ship decision criterion for my feature.”