monitoring-prediction-drift

Analyze prediction-side drift in deployed machine learning models with labeling-delay-aware windows.

2|Updated May 23, 2026

Search Tags:#machine learning #calibration #brier score #model monitoring #prediction drift #segment evaluation #label delay

One-click install

npx skills add https://github.com/rocklambros/rcs --skill monitoring-prediction-drift

Or copy as Structured Prompt for Agent▼

Please help me install this Agent Skill.
Skill: monitoring-prediction-drift
Source: https://github.com/rocklambros/rcs/tree/main/skills/ml-datasci/monitoring-prediction-drift
Command: npx skills add https://github.com/rocklambros/rcs --skill monitoring-prediction-drift

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you monitor deployed machine learning models for prediction-side drift when features look stable but outputs, calibration, or segment performance start to degrade. It prevents the common mistake of evaluating recent predictions before their labels are mature.

Core Features & Use Cases

Labeling-delay-aware evaluation: Builds reference and current windows that only include predictions with mature labels.
Calibration diagnostics: Uses reliability curves, calibration slope and intercept, and Brier score to detect miscalibration.
Per-segment monitoring: Tracks AUC, precision, recall, and F1 by cohort with bootstrap confidence intervals to catch Simpson's-paradox failures.
Decision guidance: Distinguishes calibration drift, score shift, cohort-mix change, and concept drift so you recalibrate before retraining when possible.
Use case: A loan-default model shows stable feature PSI but lower predicted scores than realized defaults, so the Skill recommends a mature-label window, calibration analysis, and post-hoc recalibration instead of a blind retrain.

Quick Start

Ask this Skill to compare your deployed model's reference and current prediction windows using mature labels, calibration diagnostics, and per-segment metrics.

Frequently Asked Questions about monitoring-prediction-drift

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I monitor machine learning prediction drift when there is a label delay?▼

Detect miscalibration in deployed models by analyzing reliability curves, calibration slope and intercept, and Brier scores to identify when predicted probabilities no longer match realized outcomes.

How do I distinguish between calibration drift and concept drift in a deployed model?▼

Track per-segment performance by calculating cohort-specific AUC, precision, recall, and F1 with bootstrap confidence intervals to detect localized degradation and avoid Simpson's paradox failures.

What is the best way to handle stable feature PSI but degrading model performance?▼

Handle stable feature PSI with degrading performance by running labeling-delay-aware calibration diagnostics to determine if post-hoc recalibration is sufficient or if a full retrain is required.

Does this prediction drift analysis apply to regression and multiclass classification systems?▼

Yes, prediction drift analysis applies to live binary, multiclass, and regression systems, evaluating score distribution shifts and calibration metrics across delayed-label production windows.