eval-trace-rca

Diagnose root causes of LLM trace failures using eval results and runtime errors.

150|23|Updated Feb 3, 2026

Search Tags:#debugging #tracing #root-cause-analysis #trace-analysis #llm-traces #eval-signals

Ready to run

npx skills add https://github.com/datadog-labs/agent-skills --skill eval-trace-rca

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-trace-rca
Source: https://github.com/datadog-labs/agent-skills/tree/main/dd-llmo/eval-trace-rca
Command: npx skills add https://github.com/datadog-labs/agent-skills --skill eval-trace-rca

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Diagnoses why production LLM traces show eval failures or runtime errors by tracing symptoms to the root cause across traces and spans.

Core Features & Use Cases

End-to-end root-cause analysis of production traces using eval signals or runtime errors.
Phase-by-phase workflow to map judge reasoning, trace context, and root causes, with actionable recommendations.
Works with ml_app or eval_name contexts to identify systemic issues affecting multiple runs.

Quick Start

Run a full RCA on a target app or eval: select ml_app or eval_name, specify timeframe, and let the tool outline the likely root causes and suggested fixes.

Frequently Asked Questions about eval-trace-rca

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I debug production LLM traces using eval signals?▼

Debug production LLM traces by applying eval judge results and runtime errors to map symptoms to root causes. The tool traces failures across spans and traces to identify systemic issues affecting multiple runs within a specified timeframe.

What is root-cause analysis for LLM trace failures?▼

Root-cause analysis for LLM trace failures is the process of diagnosing why production traces show eval failures or runtime errors. It maps judge reasoning and trace context to pinpoint the exact origin of the issue across spans.

How do I run a root-cause analysis on a specific ml_app or eval_name?▼

Run a root-cause analysis by selecting an ml_app or eval_name context and specifying a timeframe. The tool will outline the likely root causes across traces and spans, providing actionable recommendations for fixes.

Do I need runtime errors to perform LLM trace analysis?▼

You do not need runtime errors exclusively; LLM trace analysis can be performed using either eval judge results or runtime errors. The tool uses whichever signals are available to trace symptoms back to their root causes.

What data do I need to fetch span details during trace analysis?▼

To fetch span details during trace analysis, you need access to trace data, eval results, and tooling capable of retrieving content and error information. These inputs allow the tool to map symptoms to root causes effectively.