drift-monitor-designer

Start with privacy-safe data capture at the serving layer, then design four monitors: data-quality, model-quality, bias-drift, and feature-attribution-drift. Each monitor gets a baseline from training or release-gate artifacts, a schedule, thresholds, and alert actions wired to Prometheus and Alertmanager.

Use Jensen-Shannon distance or Population Stability Index for general distribution shift, Kolmogorov-Smirnov or L-infinity for numerical features, chi-squared for categorical features, and Wasserstein for continuous shape change. Match the statistical measure to the feature data type.

Yes, Evidently computes data-drift, prediction-drift, data-quality, and attribution reports and runs as a scheduled Kubernetes CronJob over captured data. Metrics flow to Prometheus, alerts fire through Alertmanager, and Grafana visualizes baseline versus current distributions.

Use a start and end offset window to merge late-arriving ground-truth labels with the correct captured predictions. Model-quality and bias monitors run as Kubernetes Jobs that join captured predictions with the delayed ground-truth store before computing metrics.

Tight thresholds on signals with unknown normal variance generate false-positive floods that train teams to ignore alerts. Start with wide conservative thresholds, match windows to temporal patterns like seasonality, and tighten only as normal variance is learned.

Model monitoring tracks the four drift signals on data, quality, bias, and attribution, while infrastructure monitoring tracks GPU, CPU, memory, and network via Prometheus and DCGM. Only infrastructure metrics drive autoscaling; conflating the two buries model-quality signals under resource noise.