skills.rest
kalilurrahman
kalilurrahmanCOMMUNITY
Explore all skills by this author →
Website𝕏@krahmanGitHub

model-monitoring

Monitor production ML models for drift, performance degradation, and data quality issues.

1|Updated Mar 21, 2026
Search Tags:#prometheus#a/b testing#mlflow#data drift#concept drift#ml monitoring#model retraining
One-click install
npx skills add https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts --skill model-monitoring-kalilurrahman
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: model-monitoring
Source: https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts/tree/main/04-ai-ml/model-monitoring
Command: npx skills add https://github.com/kalilurrahman/kr-claudiator-skills-original-prompts --skill model-monitoring-kalilurrahman

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Machine learning models degrade silently in production as input data distributions shift and real-world patterns change, but teams often lack visibility until business metrics suffer. This Skill provides a complete monitoring framework to detect drift, track performance, and trigger retraining before failures impact users. ## Core Features & Use Cases - Drift Detection: Detect data drift using Kolmogorov-Smirnov tests and Population Stability Index (PSI), plus concept drift via rolling accuracy tracking against baselines. - Prediction Logging & Metrics: Log predictions to S3 in partitioned Parquet files and expose accuracy, latency, and throughput metrics through Prometheus and Grafana. - Automated Retraining & Safe Rollouts: Trigger MLflow-based retraining pipelines when drift exceeds thresholds, and validate new models with A/B testing and shadow mode deployment. - Use Case: A fraud detection model's accuracy drops from 92% to 84% as attacker behavior changes. The monitoring pipeline detects the concept drift, alerts the team via Prometheus rules, and automatically triggers retraining on the last 90 days of data. ## Quick Start Set up production monitoring for my XGBoost fraud detection model with drift detection, Prometheus latency metrics, and automated retraining triggers.

Frequently Asked Questions about model-monitoring

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect data drift in production ML models?▼

Use the Kolmogorov-Smirnov test to compare feature distributions between baseline training data and recent production data, flagging drift when the p-value falls below 0.05. Population Stability Index (PSI) offers an alternative where scores above 0.2 indicate significant distribution change.

How to monitor ML model performance degradation over time?▼

Track accuracy over daily windows using logged predictions joined with ground truth labels, then compute a rolling 7-day average. Alert when the drop from baseline accuracy exceeds a threshold such as 5%, which indicates concept drift requiring retraining.

What is the difference between data drift and concept drift?▼

Data drift means input feature distributions changed, detected with statistical tests like KS or PSI without needing labels. Concept drift means the relationship between inputs and outputs changed, detected only by comparing predictions against ground truth over time.

Can I test a new ML model without affecting production users?▼

Yes, use shadow mode where the new model runs alongside the production model and logs its predictions without serving them to users. Alternatively, use A/B testing with consistent hash-based traffic splitting to compare accuracy on a small percentage of live traffic.

Why is ground truth required for concept drift detection?▼

Concept drift measures actual prediction correctness, which is impossible without labeled outcomes to compare against predictions. Without ground truth, you can only detect input distribution changes via data drift, not true performance degradation.

How do I set up automated ML model retraining triggers?▼

Build a scheduled pipeline that checks drift metrics daily and initiates retraining when the maximum KS statistic exceeds a threshold like 0.2 or accuracy drops significantly. Log the run with MLflow, evaluate on test data, and deploy only if the new model outperforms the current one.

Related Skills

Explore More Data & Analytics Skills→
Nir-BhayNir-Bhay
19

ml-api-endpoint

Deploy machine learning models as FastAPI inference endpoints with Dockerfile support.

Community
Advanced
DazlarusDazlarus

ml-ops

Implements model registries, serving APIs, drift detection, and automated retraining pipelines for ML operations.

Community
Advanced
pantheon-orgpantheon-org
9

promql-generator

Generates PromQL queries for metrics, alerts, and SLO burn-rate.

Official
Advanced
fmindfmind
1.4k

MLOps Observability

Configure ML observability with MLflow, Evidently, and SHAP for drift detection and alerting.

Community
Advanced