skills.rest
Diploma-pending
Diploma-pendingOFFICIAL
Explore all skills by this author →
GitHub

LangSmith Datasets

Generate and upload evaluation datasets from LangSmith trace JSONL files.

Updated Feb 24, 2026
Search Tags:#testing#evaluation#data generation#traces#datasets#langsmith
One-click install
npx skills add https://github.com/Diploma-pending/test-case --skill langsmith-datasets
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: LangSmith Datasets
Source: https://github.com/Diploma-pending/test-case/tree/main/.agents/skills/langsmith-datasets
Command: npx skills add https://github.com/Diploma-pending/test-case --skill langsmith-datasets

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires langsmith, click, rich, python-dotenv, commander, chalk, cli-table3, dotenv, and includes scripts (resource) components.

What problem does it solve?

This Skill streamlines the process of creating, managing, and uploading evaluation datasets for testing and validation, directly from exported trace files.

Core Features & Use Cases

  • Dataset Generation: Automatically create datasets in various formats (final_response, single_step, trajectory, RAG) from LangSmith trace exports.
  • LangSmith Upload: Seamlessly upload generated datasets to your LangSmith workspace for organized evaluation.
  • Use Case: After running experiments, export your traces and use this Skill to generate a final_response dataset, then upload it to LangSmith as "My Experiment Results" for easy comparison and analysis.

Quick Start

Use the langsmith-datasets skill to generate a final_response dataset from the traces in the './traces' directory and upload it to LangSmith with the name 'My Skill Datasets'.

Frequently Asked Questions about LangSmith Datasets

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create evaluation datasets from LangSmith trace files?▼

To create evaluation datasets from LangSmith trace files, you generate them from exported JSONL trace files into formats like final_response, single_step, trajectory, or RAG using automated extraction and field mapping scripts.

What types of datasets can I generate from LangSmith traces?▼

You can generate four types of datasets from LangSmith traces: final_response, single_step, trajectory, and RAG. Each type supports specific field extraction options tailored for different LLM testing and evaluation scenarios.

How do I upload generated datasets to my LangSmith workspace?▼

You upload generated datasets to your LangSmith workspace by providing your LangSmith API key in the environment. The process supports direct upload to organize your evaluation data for comparison and analysis.

Do I need a specific environment to generate LangSmith evaluation datasets?▼

Yes, you need a Python or TypeScript environment configured with your LangSmith API key. This setup is required to parse JSONL trace files and authenticate the direct upload to your LangSmith workspace.

Can I extract specific fields when creating RAG datasets from trace exports?▼

Yes, you can extract specific fields when creating RAG datasets from trace exports. The dataset generation process includes field extraction options to ensure your RAG evaluation data meets your testing requirements.

Related Skills

Explore More Software Engineering Skills→
langchain-ailangchain-ai
1.1k

LangSmith Dataset

Generate evaluation datasets from exported LangSmith trace JSONL files.

Official
Intermediate
anukkrit149anukkrit149

langsmith-dataset

Create and manage LangSmith evaluation datasets using the langsmith CLI.

Community
Intermediate
KappTech88KappTech88

langsmith-observability

Diagnose LLM behavior across tracing, evaluation, and monitoring workflows in LangSmith.

Community
Advanced
surus-latsurus-lat
8

synthesize-data

Generate synthetic benchmark datasets as JSONL input-output pairs from task specifications.

Official
Intermediate