trustworthy-experiments

To run trustworthy A/B tests and avoid false positives, apply a framework of pre-registration, power analysis, and SRM checks. This ensures your product experiments yield reliable, actionable results rather than being misled by common validity threats.

An SRM check, or Sample Ratio Mismatch check, validates that observed data allocation matches your experiment design. You need it before interpreting A/B test results to ensure the data is not skewed by instrumentation errors or bot traffic.

To estimate sample size for controlled experiments, use power analysis to achieve adequate sensitivity. Running a script like sample_size.py on your planned metrics calculates the required runtime and participant volume.

To set up guardrail metrics for feature rollout validation, define an Overall Evaluation Criterion (OEC) that balances success metrics with protective guardrails. Pre-register this plan to monitor long-term harm and prevent false positives during gradual rollouts.

Yes, you can use this framework for gradual rollouts and post-launch validation. It supports applying controlled experiments, feature pilots, and rollout studies across product lines and platforms to ensure results remain reliable.

A/B tests show significant results but fail to replicate due to inadequate power analysis, missing guardrail monitoring, or ignored SRM checks. Applying pre-registration and replication guidelines prevents these false positives.