dt-obs-genai

What problem does it solve?

This Skill provides in-depth analysis of AI Observability signals from customer GenAI applications, using DQL to identify and address issues with golden signals, LLM signals, token and cost analytics, agent signals, conversation analytics, guardrails, and evaluation quality.

Core Features & Use Cases

• Golden Signals Analysis: Monitor traffic, errors, latency, and saturation for GenAI applications.
• LLM Signals Analysis: Track LLM performance, provider latency, model performance, and token consumption.
• Cost and Tokens Analysis: Estimate costs based on token consumption and identify cost spikes.
• Agent Signals Analysis: Monitor tool usage, failing agents, step latency, loop/runaway detection, and Smartscape topology.
• Conversation Analytics: Analyze conversation sessions, cost per conversation, and session error rate.
• Guardrails and Safety: Monitor blocked and truncated responses, content filtering, and safety evaluations.
• Evaluation Quality: Assess quality scores, failed evaluations, and root-cause analysis for failed responses.
• Use Case: Consider a scenario where an application is experiencing token spikes. Use this Skill to identify the highest-spend model-provider combinations, detect token-burn spikes, and estimate costs based on token usage.

Quick Start

Use the dt-obs-genai skill to analyze the token usage by model for the last 24 hours.