langsmith-dataset

To generate evaluation datasets from LangSmith traces, use this Skill to extract data directly from your LangSmith project and upload it. It automates creating datasets like final_response, single_step, trajectory, and rag types for testing.

You can create four types of datasets from LangChain agent traces: final_response, single_step, trajectory, and rag. These are extracted from existing LangSmith traces to evaluate different aspects of agent performance.

You can filter LangSmith traces to build targeted test datasets by specifying the root run name, time range, and depth. This allows you to extract specific agent interactions for your evaluation sets.

Yes, you can upload generated datasets directly to your LangSmith project. The Skill integrates directly with the LangSmith API for seamless dataset management after extracting the data from traces.

You need specific environment variables for authentication to integrate with the LangSmith API. The Skill uses python-dotenv to load these credentials, ensuring secure access to your LangSmith traces and dataset uploads.

Use trajectory datasets to evaluate the full sequence of agent actions, and single step datasets to evaluate individual steps. Both are extracted from LangSmith traces, allowing you to target specific conversational accuracy or component performance.