craft-experiment-design

What problem does it solve?

Experiment Design helps you go from "we should test this" to a well-structured experiment design that your team and data scientists can review.

Core Features & Use Cases

• Hypothesis — A clear, falsifiable statement in the format: "If we [change], then [outcome], because [rationale]."
• Primary Metric — The single metric that determines success or failure.
• Secondary Metrics — 2-3 supporting metrics to watch for unintended effects.
• Guardrail Metrics — Metrics that must not degrade (e.g., error rates, latency, retention).
• Audience & Allocation — Who should be in the test? What percentage split do you recommend?
• Holdout Strategy — Should we maintain a holdout group after the test? Why or why not?
• Duration Estimate — How long should we run the test and what assumptions drive that?
• Risks & Considerations — What could go wrong or bias the results?

Quick Start

Draft a complete experiment plan with a falsifiable hypothesis, primary metric, secondary metrics, guardrails, audience allocation, a holdout decision, and an estimated duration.