product-experiments

Start with a falsifiable hypothesis about user behavior, define a primary success metric plus guardrail metrics, and calculate the required sample size before launching. The Skill provides frameworks like the Overall Evaluation Criterion and risk assessment questions to structure the test.

Skip experiments when results would take too long to reach significance, when the change is a low-risk industry best practice, or when launching something entirely new with no control group. The Skill's Three Reasons to Skip an Experiment framework walks through each scenario.

A Sample Ratio Mismatch occurs when the actual control-versus-treatment split deviates from the designed ratio, signaling bugs or data pipeline issues. It is checked with a chi-squared test and is considered the single most important validity check for any A/B test.

Detecting a 5% relative change on a step converting at 10% requires over 60,000 users per variation. Early-stage products can lower the confidence interval to 85% or use techniques like CUPED to reduce the sample size needed.

Evaluate three options: iterate once more with specific improvements, kill the project and reallocate resources, or ship anyway and claw back the negative impact. Failed experiments still provide learning value when hypotheses and insights are documented.

Twyman's Law states that any result that looks too good to be true usually is, with flaws found roughly nine out of ten times. Investigate suspicious wins by checking for sample ratio mismatches, data pipeline errors, and replicating the experiment before celebrating.