skills.rest
doanchienthangdev
doanchienthangdevCOMMUNITY
Explore all skills by this author →
GitHub

Model Monitoring

Detect data drift and track performance metrics in deployed machine learning models.

4|1|Updated Dec 30, 2025
Search Tags:#mlops#alerting#performance monitoring#explainability#data drift#model monitoring
One-click install
npx skills add https://github.com/doanchienthangdev/omgkit --skill model-monitoring
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Model Monitoring
Source: https://github.com/doanchienthangdev/omgkit/tree/main/plugin/skills/ai-ml/model-monitoring
Command: npx skills add https://github.com/doanchienthangdev/omgkit --skill model-monitoring

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires scipy, numpy, evidently, prometheus_client, pandas, sklearn, requests, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill addresses the critical need to ensure Machine Learning models remain accurate and reliable in production by detecting and alerting on issues like data drift and performance degradation.

Core Features & Use Cases

  • Data Drift Detection: Identifies changes in input data distributions using statistical tests (Kolmogorov-Smirnov, Chi-squared, PSI) and libraries like Evidently AI.
  • Performance Monitoring: Tracks key metrics (accuracy, precision, recall, latency) and compares them against baselines.
  • Alerting System: Configures rules to notify stakeholders via channels like Slack when predefined thresholds for drift or performance drops are breached.
  • Use Case: Automatically monitor a deployed fraud detection model. If the distribution of transaction features changes significantly (data drift) or the model's accuracy drops below 90% (performance degradation), the system triggers a critical alert to the MLOps team.

Quick Start

Use the Model Monitoring skill to detect data drift in the production data compared to the training data.

Frequently Asked Questions about Model Monitoring

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect data drift in machine learning models?▼

Detect data drift by comparing production data distributions against training baselines using statistical tests like Kolmogorov-Smirnov, Chi-squared, and PSI. This identifies significant changes in input features that may degrade model performance.

How does performance monitoring work for deployed ML models?▼

Performance monitoring tracks key metrics like accuracy, precision, recall, and latency. It continuously compares these metrics against established baselines to identify and alert on significant model degradation in production environments.

Can I set up Slack alerting when model accuracy drops?▼

Yes, you can configure alerting rules to send notifications via channels like Slack. Alerts trigger automatically when predefined thresholds for data drift or performance metric drops are breached.

What is the best way to monitor a fraud detection model in production?▼

Monitor a fraud detection model by tracking transaction feature distributions for data drift and setting performance thresholds. If accuracy drops below baseline limits, the system triggers critical alerts to the MLOps team.

Does this model monitoring approach work with Evidently AI and pandas?▼

Yes, this approach integrates with Evidently AI and pandas for data processing. It leverages these libraries alongside scipy and sklearn to execute statistical drift detection and performance tracking workflows.

What statistical tests are used for detecting input data distribution changes?▼

Statistical tests for detecting input data distribution changes include Kolmogorov-Smirnov, Chi-squared, and Population Stability Index (PSI). These tests compare current production data against training data baselines to identify drift.

Related Skills

Explore More Data & Analytics Skills→
crossxwillcrossxwill
5

ydata-eda-profiling

Generate reproducible ydata-profiling EDA reports from pandas DataFrames.

Community
Intermediate
devendrapratapsinghdevendrapratapsingh
1

mlops

Automate ML model deployment, training pipelines, and monitoring with Kubeflow, MLflow, Feast, and Seldon Core.

Community
Advanced
fmindfmind
1.4k

MLOps Observability

Configure ML observability with MLflow, Evidently, and SHAP for drift detection and alerting.

Community
Advanced
TrevorMannTrevorMann

Data Quality Triage

Route records to done, needs_review, or unsalvageable based on completeness and confidence signals.

Community
Intermediate