skills.rest
HermeticOrmus
HermeticOrmusCOMMUNITY
Explore all skills by this author →
Website𝕏@dbodartmGitHub

model-monitoring-patterns

Detects data and concept drift in production ML models using Evidently, NannyML, whylogs, and Prometheus.

Updated Feb 25, 2026
Search Tags:#MLOps#alerting#data drift#model monitoring#concept drift#performance estimation
One-click install
npx skills add https://github.com/HermeticOrmus/LibreMLOps-Claude-Code --skill model-monitoring-patterns
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: model-monitoring-patterns
Source: https://github.com/HermeticOrmus/LibreMLOps-Claude-Code/tree/main/plugins/model-monitoring/skills/model-monitoring-patterns
Command: npx skills add https://github.com/HermeticOrmus/LibreMLOps-Claude-Code --skill model-monitoring-patterns

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pandas, numpy, scipy, evidently, nannyml, whylogs, prometheus_client, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the critical need to monitor machine learning models in production for performance degradation, data drift, and concept drift, ensuring models remain reliable and accurate over time.

Core Features & Use Cases

  • Data Drift Detection: Identifies shifts in input data distributions using metrics like PSI, KS-test, and Wasserstein distance.
  • Performance Estimation: Estimates model performance (e.g., AUC, F1) even when ground truth labels are delayed, using techniques like NannyML's CBPE.
  • Alerting & Reporting: Generates reports and alerts for significant drift or performance drops, integrating with tools like Evidently and Prometheus.
  • Use Case: Automatically detect when the distribution of user demographics in your recommendation system changes significantly, potentially impacting model accuracy, and trigger an alert for investigation.

Quick Start

Use the model monitoring patterns skill to generate a drift report comparing reference data to current production data.

Frequently Asked Questions about model-monitoring-patterns

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor machine learning models for data drift in production?▼

To monitor models for data drift in production, you compare current feature distributions against a baseline using statistical metrics like PSI, KS-test, and Wasserstein distance to detect significant changes.

What is concept drift and how can I detect it without ground truth labels?▼

Concept drift occurs when the relationship between features and target variables changes. You can detect it without ground truth labels by using performance estimation techniques like NannyML's CBPE to estimate metrics such as AUC or F1.

Can I use Evidently and NannyML together for MLOps performance estimation?▼

Yes, you can integrate Evidently and NannyML together for MLOps performance estimation, using Evidently for comprehensive drift analysis reports and NannyML for estimating model metrics without ground truth.

What's the best way to set up alerting for model drift and performance drops?▼

The best way to set up alerting for model drift and performance drops is to integrate drift detection libraries like Evidently and whylogs with Prometheus to track operational metrics and trigger alerts automatically.

Do I need pandas and numpy to generate a drift report comparing reference data?▼

Yes, you need pandas and numpy to handle data structures and numerical operations required to generate a drift report comparing your reference data to current production data.

Related Skills

Explore More Data & Analytics Skills→
StarlitnightlyStarlitnightly
1.2k

data-stats-analysis

Perform statistical tests and corrections on datasets using SciPy, statsmodels, and NumPy.

Community
Advanced
yonatangrossyonatangross
217

monitoring-observability

Implement Prometheus metrics, Grafana dashboards, and Langfuse tracing for application monitoring.

Community
Advanced
profmikegreeneprofmikegreene
1

data-exploration

Profile and analyze datasets to surface data quality issues and insights.

Community
Advanced
doanchienthangdevdoanchienthangdev
4

robust-ai

Monitor production ML models and detect data or concept drift.

Community
Advanced