skills.rest
dotlab-hq
dotlab-hqOFFICIAL
Explore all skills by this author →
GitHub

langsmith-evaluator

Creates LangSmith evaluation pipelines automating grading of agent and LLM outputs.

Updated Apr 12, 2026
Search Tags:#cli#langchain#evaluation#trajectory#langsmith#llm-judge#evaluator
One-click install
npx skills add https://github.com/dotlab-hq/torque --skill langsmith-evaluator-dotlab-hq
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: langsmith-evaluator
Source: https://github.com/dotlab-hq/torque/tree/main/.agents/skills/langsmith-evaluator
Command: npx skills add https://github.com/dotlab-hq/torque --skill langsmith-evaluator-dotlab-hq

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill provides a practical blueprint for building repeatable evaluation pipelines that standardize and automate grading of agent and LLM outputs within LangSmith projects and datasets.

Core Features & Use Cases

  • Evaluator Patterns: Guidance for both LLM-as-judge evaluators and deterministic custom code evaluators for objective checks.
  • Run Function Design: Instructions to capture outputs and execution trajectories, handle RunTree vs dict run objects, and align run outputs to dataset schema.
  • Deployment & Workflow: How to run evaluations locally with evaluate(), upload code evaluators via the langsmith CLI, attach evaluators to datasets or projects, and handle sandbox constraints.
  • Use Case: Validate a conversational agent's final answers and tool-call trajectories against an offline dataset or perform live quality checks on production runs.

Quick Start

Run a local evaluation by calling evaluate with your run function and a dataset named My Dataset to test evaluators before uploading.

Frequently Asked Questions about langsmith-evaluator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a LangSmith evaluation pipeline for agent trajectories?▼

To build a LangSmith evaluation pipeline, design a run function that captures outputs and execution trajectories, ensuring you handle RunTree versus dict run objects and align run outputs to your dataset schema for automated grading.

How do I test LangSmith evaluators locally before uploading?▼

You test LangSmith evaluators locally by calling the evaluate() function with your run function and a target dataset, allowing you to validate evaluator logic before uploading code evaluators via the langsmith CLI.

Can I use LangSmith evaluators for live quality checks on production agent runs?▼

Yes, you can apply LangSmith evaluators to perform online project-level quality checks for agents implemented in Python or TypeScript, standardizing the grading of both final answers and tool-call trajectories.

Do I need LangSmith API authentication to run evaluation pipelines?▼

LangSmith evaluation pipelines require LangSmith API authentication to function, which is necessary for both offline dataset-based evaluations and uploading custom code evaluators to the sandbox environment.

What is the difference between LLM-as-judge and custom code evaluators in LangSmith?▼

LLM-as-judge evaluators grade subjective agent outputs using language models, while deterministic custom code evaluators perform objective checks, with both patterns requiring single-metric evaluator returns to function correctly.

Why does my uploaded LangSmith evaluator fail in the sandbox?▼

Uploaded LangSmith evaluators fail when they violate sandbox constraints or return multiple metrics instead of enforcing a single-metric evaluator return, requiring alignment with the uploaded evaluator sandbox environment.

Related Skills

Explore More Software Engineering Skills→
lguibrlguibr

langsmith-evaluator

Evaluate agent outputs with LangSmith evaluators and run functions.

Community
Advanced
wpsadiwpsadi

langsmith-evaluator

Build LangSmith evaluation pipelines with LLM-as-Judge and custom code.

Community
Advanced
anukkrit149anukkrit149

langsmith-evaluator

Build LangSmith evaluator functions and run logic for agent output scoring.

Community
Advanced
jxtngxjxtngx
2

langsmith-evaluator

Create and deploy LLM agent evaluation pipelines within the LangSmith ecosystem.

Community
Advanced