auditing-embedding-drift

What problem does it solve?

This Skill helps teams determine whether retrieval quality is slipping because embedding distributions have changed, instead of guessing from a single metric or a raw recall drop.

Core Features & Use Cases

• Compares baseline and comparison embedding cohorts using per-dimension Jensen-Shannon divergence, centroid cosine distance, and intra-cohort spread change.
• Bootstraps confidence intervals so sample noise is not mistaken for actionable drift.
• Attributes drift to likely causes such as provider model re-versioning, new content categories, or upstream data changes.
• Use it when a long-running RAG system degrades after months of stable code, after an embedding provider update, or when comparing production slices across time, region, or source.

Quick Start

Ask the skill to audit two embedding cohorts from your retrieval system and report the drift metrics, confidence intervals, likely cause, and recommended action.