monitor-ai-quality

Use this Skill to query Amplitude Agent Analytics metrics including quality scores, success rates, sentiment, and rubric scores. It runs parallel queries for overview metrics, time series trends, and recent failed sessions, then compiles a health report with trend detection and agent rankings.

The Skill queries cost and performance metrics including total cost, cost time series, and latency percentiles, then breaks them down per agent. For cost anomalies, it can query spans grouped by model name to show which models and tools drive spending.

No. The Skill requires Amplitude Agent Analytics to be instrumented in your project, since all data comes from the Amplitude MCP agent analytics endpoints. If no AI session data exists, it reports that instrumentation is missing and suggests checking the AI agent SDK integration.

Quality score below 0.4, success rate below 60%, sentiment below 0.5, task failure rate above 25%, or P90 latency above 30 seconds are flagged as critical. Warning ranges sit between those and the good thresholds, such as quality between 0.4 and 0.7.

If fewer than 50 sessions exist in the window, the Skill notes that sample sizes are small and findings may not be statistically meaningful. It recommends extending the time window to gather more data before drawing conclusions.