trustworthy-experiments

Plan and run A/B tests with power analysis and SRM checks.

18|3|Updated Jan 15, 2026

Search Tags:#A/B testing #replication #guidelines #experiments #power analysis #trustworthy #SRM

Ready to run

npx skills add https://github.com/wdavidturner/product-skills --skill trustworthy-experiments

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: trustworthy-experiments
Source: https://github.com/wdavidturner/product-skills/tree/main/skills/trustworthy-experiments
Command: npx skills add https://github.com/wdavidturner/product-skills --skill trustworthy-experiments

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Trustworthy Experiments provides a framework to design, run, and interpret controlled experiments (A/B tests) so results are reliable, actionable, and not misled by common validity threats.

Core Features & Use Cases

Planning and preregistration with a clear Evaluation Criterion (OEC) balancing success and guardrail metrics.
Power analysis, sample-size estimation, and runtime guidance to achieve adequate sensitivity.
SRM checks, replication, and guardrail monitoring to prevent false positives and long-term harm.
Use Cases: A/B tests, feature pilots, gradual rollouts, and post-launch validation across product lines.

Quick Start

Use the included references and scripts to design a pre-registered experiment plan: fill out an experiment plan with the template, run sample_size.py for required sample size, and run srm_check.py on observed data to validate SRM before interpreting results.

Frequently Asked Questions about trustworthy-experiments

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run trustworthy A/B tests and avoid false positives?▼

To run trustworthy A/B tests and avoid false positives, apply a framework of pre-registration, power analysis, and SRM checks. This ensures your product experiments yield reliable, actionable results rather than being misled by common validity threats.

What is an SRM check and when do I need it for experiment evaluation?▼

An SRM check, or Sample Ratio Mismatch check, validates that observed data allocation matches your experiment design. You need it before interpreting A/B test results to ensure the data is not skewed by instrumentation errors or bot traffic.

How do I estimate sample size for controlled experiments?▼

To estimate sample size for controlled experiments, use power analysis to achieve adequate sensitivity. Running a script like sample_size.py on your planned metrics calculates the required runtime and participant volume.

How do I set up guardrail metrics for feature rollout validation?▼

To set up guardrail metrics for feature rollout validation, define an Overall Evaluation Criterion (OEC) that balances success metrics with protective guardrails. Pre-register this plan to monitor long-term harm and prevent false positives during gradual rollouts.

Can I use this framework for gradual rollouts and post-launch validation?▼

Yes, you can use this framework for gradual rollouts and post-launch validation. It supports applying controlled experiments, feature pilots, and rollout studies across product lines and platforms to ensure results remain reliable.

Why does my A/B test show significant results but fail to replicate?▼

A/B tests show significant results but fail to replicate due to inadequate power analysis, missing guardrail monitoring, or ignored SRM checks. Applying pre-registration and replication guidelines prevents these false positives.