langsmith-trace-analyzer

You can analyze LangSmith traces by running a Python-based script against a directory of downloaded traces organized by outcome. This generates diagnostic reports covering token usage, message patterns, and tool calls to identify failures.

To generate debugging reports, process your organized trace directories to compare passed versus failed runs. The analysis produces summary statistics and pattern comparisons outputted in Markdown or JSON format for benchmarking.

Yes, you can extract metrics like message counts, token usage, tool calls, and duration from raw LangSmith trace data. The analysis tolerates multiple trace payload shapes to consistently produce these summary statistics.

The best way to summarize inconsistent LangSmith trace payloads is using a tolerant analysis script that handles multiple payload shapes. It extracts patterns and metrics to produce clear Markdown or JSON analysis reports.

Yes, you need to organize downloaded LangSmith traces by outcome status such as passed, failed, or error before analysis. This categorization allows the script to accurately generate comparisons and diagnostics for your reports.