ds-metric-validation

Metric validation verifies numerator and denominator definitions, unit alignment, and missingness before reporting A/B testing lift. It enforces minimum definition checks and escalates into robustness diagnostics when SRM, denominator drift, or leakage are suspected.

Denominator drift distorts experiment results by shifting the baseline against which treatment lift is measured. Validation tools detect this by applying escalation triggers that identify distributional instability and segment concentration before results reach stakeholders.

SRM, or Sample Ratio Mismatch, indicates a systematic imbalance in experiment arm sizes. You need invariant checks during A/A validation when SRM failures occur, triggering standard diagnostics to assess unit alignment and logging artifacts.

Checking for post-treatment leakage and outlier dominance involves running robustness diagnostics that quantify outlier contribution. Validation workflows test alternative metric definitions and assess sensitivity to operational artifacts to ensure lift remains decision-safe.

Yes, metric validation applies to metric-driven reporting workflows beyond active A/B tests. It ensures that numerator and denominator logic, filter logic, and null handling remain interpretable and stable for any analytics reporting.

Minimum metric checks only verify baseline definitions, filter logic, and unit alignment. They lack the robustness diagnostics needed to detect sign flips, outlier dominance, and missingness sensitivity, which can silently break experiment conclusions.