eval-trace-rca

Debug production LLM traces by applying eval judge results and runtime errors to map symptoms to root causes. The tool traces failures across spans and traces to identify systemic issues affecting multiple runs within a specified timeframe.

Root-cause analysis for LLM trace failures is the process of diagnosing why production traces show eval failures or runtime errors. It maps judge reasoning and trace context to pinpoint the exact origin of the issue across spans.

Run a root-cause analysis by selecting an ml_app or eval_name context and specifying a timeframe. The tool will outline the likely root causes across traces and spans, providing actionable recommendations for fixes.

You do not need runtime errors exclusively; LLM trace analysis can be performed using either eval judge results or runtime errors. The tool uses whichever signals are available to trace symptoms back to their root causes.

To fetch span details during trace analysis, you need access to trace data, eval results, and tooling capable of retrieving content and error information. These inputs allow the tool to map symptoms to root causes effectively.