monitoring-prediction-drift

What problem does it solve?

This Skill helps you monitor deployed machine learning models for prediction-side drift when features look stable but outputs, calibration, or segment performance start to degrade. It prevents the common mistake of evaluating recent predictions before their labels are mature.

Core Features & Use Cases

• Labeling-delay-aware evaluation: Builds reference and current windows that only include predictions with mature labels.
• Calibration diagnostics: Uses reliability curves, calibration slope and intercept, and Brier score to detect miscalibration.
• Per-segment monitoring: Tracks AUC, precision, recall, and F1 by cohort with bootstrap confidence intervals to catch Simpson's-paradox failures.
• Decision guidance: Distinguishes calibration drift, score shift, cohort-mix change, and concept drift so you recalibrate before retraining when possible.
• Use case: A loan-default model shows stable feature PSI but lower predicted scores than realized defaults, so the Skill recommends a mature-label window, calibration analysis, and post-hoc recalibration instead of a blind retrain.

Quick Start

Ask this Skill to compare your deployed model's reference and current prediction windows using mature labels, calibration diagnostics, and per-segment metrics.