LangSmith Dataset

To create an evaluation dataset from LangSmith traces, you can use a script to parse exported JSONL trace files and automatically extract inputs and outputs into structured formats for testing your AI applications.

You can generate multiple dataset types from exported trace files, including final_response for evaluating overall performance, single_step, trajectory for analyzing tool-use patterns, and RAG datasets for retrieval validation.

Yes, you can directly upload your generated datasets to your LangSmith account for centralized management and analysis after processing your exported JSONL trace files.

You need a Python or TypeScript environment with specified dependencies like langsmith, click, and rich installed to run the scripts that process your trace files and generate datasets.

The best way to analyze agent tool-use patterns is to generate a trajectory dataset from your exported LangSmith traces, which structures the sequential tool calls and actions for evaluation.

Generating a final_response dataset from traces streamlines AI testing by automatically extracting the agent's final outputs from JSONL trace files, allowing you to evaluate overall performance without manual data entry.