experimental-design

Name: experimental-design
Availability: InStock
Author: Anduril

Design experiments with pre-registered metrics, sample sizing, and validity mitigation.

7|2|Updated Mar 5, 2026

Search Tags:#hypothesis testing #causal inference #a/b testing #power analysis #pre-registration #experimental design #threats to validity

Ready to run

npx skills add https://github.com/AndurilCode/craftwork --skill experimental-design-andurilcode

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: experimental-design
Source: https://github.com/AndurilCode/craftwork/tree/main/skills/experimental-design
Command: npx skills add https://github.com/AndurilCode/craftwork --skill experimental-design-andurilcode

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps teams prevent false confidence by designing experiments that define evidence, test assumptions, and distinguish the claimed effect from alternative explanations.

Core Features & Use Cases

Pre-registration of success criteria: Specifies primary/secondary metrics, guardrails, and decision rules before collecting data to avoid “garden of forking paths.”
Robust experimental design selection: Chooses and justifies RCT/A-B testing or quasi-experimental alternatives (difference-in-differences, regression discontinuity, interrupted time series) matched to feasibility.
Threats-to-validity planning: Identifies internal, external, and construct validity risks and sets mitigation/residual risk so conclusions are interpretable.

Quick Start

Design an experiment to validate the hypothesis that Feature X increases 30-day retention by at least 2 percentage points, including variables, a recommended design, required sample size, and pre-registered decision rules.

Frequently Asked Questions about experimental-design

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I design an A/B test to validate a hypothesis without falling into the garden of forking paths?▼

To design a valid A/B test, pre-register your primary and secondary metrics, guardrails, and decision rules before collecting data. This prevents the garden of forking paths by specifying success criteria and the statistical analysis plan upfront.

What's the best way to calculate sample size and power analysis for an experimental design?▼

The best way to calculate sample size is through power analysis applied during experimental design. This process determines the required sample size to reliably detect the claimed effect, ensuring the test is appropriately powered before data collection begins.

How do I mitigate threats to validity when running quasi-experiments?▼

Mitigate threats to validity in quasi-experiments by explicitly identifying internal, external, and construct validity risks during the design phase. The plan sets specific mitigation strategies and documents residual risk so conclusions remain interpretable.

When should I use difference-in-differences or regression discontinuity instead of a randomized controlled trial?▼

Use difference-in-differences, regression discontinuity, or interrupted time series instead of an RCT when randomization is not feasible. The selection depends on your operational constraints, and the chosen quasi-experimental design must be explicitly justified.

Can I pre-register success metrics and guardrail metrics for causal inference testing?▼

Yes, you can and should pre-register success and guardrail metrics for causal inference testing. Pre-registration defines the primary metrics and decision rules before evaluation, preventing false confidence and distinguishing the claimed effect from alternatives.

Why does my experiment conclude a feature worked when alternative explanations still exist?▼

An experiment might falsely conclude a feature worked if threats to validity were not mitigated. Explicitly planning internal, external, and construct validity risks during experimental design distinguishes the true causal effect from alternative explanations.