ml-monitoring

Use statistical tests comparing current production data against a reference training baseline. PSI works well for continuous features with thresholds at 0.1 and 0.2, KS tests measure distribution distance, and chi-squared tests handle categorical features.

Data drift means the input feature distribution changed while the underlying relationship stays the same. Concept drift means the relationship between inputs and outputs changed, so the same inputs now produce wrong predictions even with stable input distributions.

PSI below 0.1 indicates a stable distribution, 0.1 to 0.2 indicates slight drift warranting a warning, and above 0.2 indicates significant drift requiring critical alerts. Bins are built from the reference data histogram with smoothing to avoid zero proportions.

Track prediction distribution drift using Wasserstein distance as an early signal, then evaluate AUC, F1, precision, and recall in windows once ground truth arrives. Compare metrics against baseline and alert when relative degradation exceeds a threshold like 5 percent.

Yes, when a monitoring check returns critical status, the pipeline sends a POST request to a CI/CD retraining webhook with the drift report. Scheduled retraining is also recommended even without detected drift to keep models fresh.

Statistical tests like KS and chi-squared need sufficient samples to produce reliable p-values. The monitoring job skips checks when fewer than 100 samples are available in the window, since results would be noisy and trigger false alerts.