skills.rest
InnerScopeHearing
InnerScopeHearingOFFICIAL
Explore all skills by this author →
GitHub

embedding-drift-monitor

Compare Azure AI Search probe results against a baseline to detect embedding drift.

Updated May 30, 2026
Search Tags:#quality assurance#monitoring#search#azure ai#embedding drift
One-click install
npx skills add https://github.com/InnerScopeHearing/otchealth-claude-tools --skill embedding-drift-monitor
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: embedding-drift-monitor
Source: https://github.com/InnerScopeHearing/otchealth-claude-tools/tree/main/skills/embedding-drift-monitor
Command: npx skills add https://github.com/InnerScopeHearing/otchealth-claude-tools --skill embedding-drift-monitor

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires otchealth-kb-memory/azure-secret.mjs, and includes scripts (resource) components.

What problem does it solve?

This Skill serves as a monitor to ensure recall-quality of Azure AI Search indexes and alerts on any drift in the results, preventing silent degradations.

Core Features & Use Cases

  • Index Monitoring: Samples and scores probe queries against Azure AI Search indexes.
  • Drift Detection: Compares against a baseline to identify potential drift in results.
  • Reporting: Generates a report or emits events to alert when drift is detected.
  • Use Case: Utilizes the Skill as part of Azure GenAIOps patterns to continuously check retrieval quality of search indexes without blocking the CI process.

Quick Start

Use the drift.mjs script with --json out.json to generate a structured report on embedding drift.

Frequently Asked Questions about embedding-drift-monitor

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor recall-quality in Azure AI Search indexes?▼

Embedding drift detection compares current Azure AI Search results against a stored baseline to identify silent degradations. The Skill samples probe queries and alerts when retrieval quality drops, preventing unnoticed performance losses.

How do I detect embedding drift in Azure AI Search without blocking CI?▼

Generate a structured drift report by running the drift script with the `--json out.json` flag. This outputs a structured file containing the recall-quality comparison results against the baseline.

Do I need authentication to check Azure AI Search for embedding drift?▼

Yes, checking Azure AI Search for embedding drift requires authentication for Azure and PostHog. You must configure valid credentials to probe indexes and emit alert events when drift is detected.

How does baseline comparison work for search index drift detection?▼

Baseline comparison for search index drift detection works by storing initial recall-quality scores in a drift-baseline.json file. Subsequent probe query results are compared against this baseline to flag any drift.

Related Skills

Explore More Software Engineering Skills→
JuanMS20JuanMS20

github-auth

Configure GitHub authentication for Hermes Agent using git, gh CLI, or SSH credentials.

Community
Intermediate
PostHogPostHog
58

logs-python

Send structured Python logs to PostHog via OpenTelemetry OTLP exporter.

Official
Intermediate
microsoft-foundrymicrosoft-foundry
56

azure-ai-fine-tuning

Automate end-to-end Azure AI Foundry fine-tuning workflows with dataset prep, training, evaluation, and deployment.

Official
Advanced
RxFitRxFit

canary

Monitor live web apps for console errors and performance regressions post-deployment.

Community
Advanced