error-analysis

You can analyze LLM traces by ingesting eval JSON, agent logs, or JSONL production runs to automatically classify failures into reasoning errors, factual errors, format violations, and tool misuse. The system parses these trace formats to detect recurring patterns and quantify their frequency across runs.

Error taxonomy is a structured classification of LLM execution failures that groups similar issues like reasoning errors or tool misuse. Applied to agent logs, it identifies recurring failure patterns and generates root-cause hypotheses to guide systematic debugging and model improvements.

Yes, you can ingest eval JSON files to automatically generate an error taxonomy and root-cause hypotheses. The analysis produces a structured report with prioritized recommendations for fixing the identified failure modes across your evaluation runs.

This approach works with production JSONL logs, eval JSON, and agent logs. It parses various trace formats to classify failures and detect recurring patterns, making it suitable for both evaluation results and live production run analysis.

The best way to identify tool misuse is to aggregate execution traces from multiple runs and categorize failures automatically. This detects recurring tool misuse patterns, quantifies their frequency, and outputs root-cause hypotheses within a prioritized recommendations report.