experimental-design

To design a valid A/B test, pre-register your primary and secondary metrics, guardrails, and decision rules before collecting data. This prevents the garden of forking paths by specifying success criteria and the statistical analysis plan upfront.

The best way to calculate sample size is through power analysis applied during experimental design. This process determines the required sample size to reliably detect the claimed effect, ensuring the test is appropriately powered before data collection begins.

Mitigate threats to validity in quasi-experiments by explicitly identifying internal, external, and construct validity risks during the design phase. The plan sets specific mitigation strategies and documents residual risk so conclusions remain interpretable.

Use difference-in-differences, regression discontinuity, or interrupted time series instead of an RCT when randomization is not feasible. The selection depends on your operational constraints, and the chosen quasi-experimental design must be explicitly justified.

Yes, you can and should pre-register success and guardrail metrics for causal inference testing. Pre-registration defines the primary metrics and decision rules before evaluation, preventing false confidence and distinguishing the claimed effect from alternatives.

An experiment might falsely conclude a feature worked if threats to validity were not mitigated. Explicitly planning internal, external, and construct validity risks during experimental design distinguishes the true causal effect from alternative explanations.