LangSmith Dataset

Name: LangSmith Dataset
Availability: InStock
Author: LangChain

Generate evaluation datasets from exported LangSmith trace JSONL files.

1.1k|86|Updated Jan 22, 2026

Search Tags:#testing #evaluation #data export #langsmith #trace analysis #dataset generation

Ready to run

npx skills add https://github.com/langchain-ai/langchain-skills --skill langsmith-dataset

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: LangSmith Dataset
Source: https://github.com/langchain-ai/langchain-skills/tree/main/config/skills/langsmith-dataset
Command: npx skills add https://github.com/langchain-ai/langchain-skills --skill langsmith-dataset

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires langsmith, click, rich, python-dotenv, commander, chalk, cli-table3, dotenv, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the creation of evaluation datasets from your LangSmith traces, streamlining the process of testing and validating your AI applications.

Core Features & Use Cases

Dataset Generation: Automatically generate datasets in various formats (final_response, single_step, trajectory, RAG) from exported trace files.
LangSmith Upload: Directly upload generated datasets to your LangSmith account for centralized management and analysis.
Use Case: After running your agent on a set of prompts, export the traces and use this Skill to generate a final_response dataset to evaluate the agent's overall performance, or a trajectory dataset to analyze its tool-use patterns.

Quick Start

Use the LangSmith Dataset skill to generate a final_response dataset from the traces in the './traces' directory and save it to './datasets/final_response.json'.

Frequently Asked Questions about LangSmith Dataset

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I create an evaluation dataset from LangSmith traces?▼

To create an evaluation dataset from LangSmith traces, you can use a script to parse exported JSONL trace files and automatically extract inputs and outputs into structured formats for testing your AI applications.

What types of datasets can I generate from exported trace files?▼

You can generate multiple dataset types from exported trace files, including final_response for evaluating overall performance, single_step, trajectory for analyzing tool-use patterns, and RAG datasets for retrieval validation.

Can I directly upload generated datasets to my LangSmith account?▼

Yes, you can directly upload your generated datasets to your LangSmith account for centralized management and analysis after processing your exported JSONL trace files.

Do I need a specific environment to generate datasets from LangSmith traces?▼

You need a Python or TypeScript environment with specified dependencies like langsmith, click, and rich installed to run the scripts that process your trace files and generate datasets.

What is the best way to analyze agent tool-use patterns from LangSmith traces?▼

The best way to analyze agent tool-use patterns is to generate a trajectory dataset from your exported LangSmith traces, which structures the sequential tool calls and actions for evaluation.

Why does generating a final_response dataset from traces streamline AI testing?▼

Generating a final_response dataset from traces streamlines AI testing by automatically extracting the agent's final outputs from JSONL trace files, allowing you to evaluate overall performance without manual data entry.