online-experimentation

A/B testing during model deployment measures business impact by evaluating live metrics and KPIs with statistical rigor. It bridges the gap between offline evaluation and online performance to handle distribution shifts and novelty effects.

Online experimentation uses Welch's t-test, Benjamini-Hochberg FDR control, and CUPED for variance reduction. These advanced statistical methods analyze experimental results to interpret the effectiveness of new models accurately.

You need online experimentation when offline metrics fail to reflect actual business impact. It tackles distribution shifts, metric mismatches, and novelty effects to understand true effects on retention and revenue during feature testing.

Bandits maximize cumulative rewards through sequential decision-making in online evaluation. They provide an alternative to standard A/B testing by dynamically allocating traffic based on real-time performance feedback.

Yes, you need scipy and numpy to run statistical testing for model rollouts. These dependencies provide the necessary computational foundations to execute advanced tests and analyze experimentation results.

The best way to control false discovery rate across multiple A/B test metrics is using Benjamini-Hochberg FDR control. This statistical method adjusts p-values to minimize false positives when evaluating numerous live metrics simultaneously.