set-up-drift-alerts

What problem does it solve?

This Skill helps catch LLM app drifts in production, alerting on changes in quality, cost, latency, and inputs that could indicate degradation.

Core Features & Use Cases

• Quality Drift Alerting: Monitor changes in model performance like faithfulness and relevance.
• Cost Drift Alerting: Detect cost creep from changes in tokens or spend.
• Latency / Error Drift Alerting: Identify changes in latency or error rates.
• Input Drift Alerting: Monitor input distribution changes.
• Use Case: Set up alerts for a production LLM app to ensure its performance remains consistent with baseline metrics.

Quick Start

Set up drift alerts for your LLM app by running the 'set-up-drift-alerts' command and providing necessary monitoring metrics.