skills.rest
anukkrit149
anukkrit149COMMUNITY
Explore all skills by this author →
Website𝕏@anukkritGitHub

langsmith-evaluator

Build LangSmith evaluator functions and run logic for agent output scoring.

Updated May 26, 2026
Search Tags:#llm-as-judge#evaluation#langsmith#custom-code#trajectory-analysis#run-functions#dataset-evaluators
One-click install
npx skills add https://github.com/anukkrit149/anukkrit-skills --skill langsmith-evaluator-anukkrit149
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: langsmith-evaluator
Source: https://github.com/anukkrit149/anukkrit-skills/tree/main/cloud/skills/langsmith-evaluator
Command: npx skills add https://github.com/anukkrit149/anukkrit-skills --skill langsmith-evaluator-anukkrit149

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires langsmith, langchain-openai, python-dotenv, openai.

What problem does it solve?

LangSmith evaluation pipelines can be hard to set up correctly because evaluator functions must match the exact run output and dataset/project schema. This skill provides a practical blueprint for building evaluators and run functions that reliably grade agent performance.

Core Features & Use Cases

  • Create evaluators (LLM-as-judge and custom code patterns) to measure accuracy, relevance, and output quality with structured scoring.
  • Define run functions that capture the right outputs (and optionally trajectories/tool calls) for later evaluation.
  • Run evaluations locally or via LangSmith uploads using the LangSmith CLI for dataset (offline) and project (online) evaluators.

Quick Start

Use this skill to build an offline dataset evaluator in code that scores agent outputs against expected fields, then run it locally with evaluate() before uploading to LangSmith.

Frequently Asked Questions about langsmith-evaluator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create LangSmith evaluators to score agent outputs?▼

LangSmith evaluators grade agent outputs by defining custom functions that compare a run against expected examples, returning a structured score. You can build LLM-as-judge or custom code patterns to measure accuracy and relevance.

What is the difference between offline dataset evaluators and online project evaluators in LangSmith?▼

Offline dataset evaluators perform (run, example) comparisons for experimentation, while online project evaluators perform real-time (run) quality checks across Python and TypeScript agent stacks.

How do I run LangSmith evaluations locally before uploading?▼

You can execute LangSmith evaluations locally using the evaluate() function to test that your run functions capture the correct outputs and trajectories before uploading them via the LangSmith CLI.

Why do my LangSmith evaluator functions fail to match the run output schema?▼

LangSmith evaluators fail when run functions do not correctly handle run output shapes or enforce one-metric-per-evaluator returns. Ensure your custom code matches the exact dataset and project schema.

Can I use LangSmith evaluators with TypeScript agent stacks?▼

Yes, LangSmith evaluators apply to both Python and TypeScript agent stacks, allowing you to score offline dataset comparisons and online project quality checks across different environments.

Do I need the LangSmith CLI to upload custom code evaluators?▼

Yes, the LangSmith CLI is used to upload custom code evaluators for both offline dataset and online project evaluations after you validate your run functions locally with evaluate().

Related Skills

Explore More Software Engineering Skills→
jackjin1997jackjin1997
11

langsmith-evaluator

Create and deploy custom evaluation metrics for AI models on LangSmith.

Community
Intermediate
joonlabjoonlab
8

langsmith-evaluator

Create and deploy LangSmith evaluation pipelines for LLM agents.

Community
Advanced
aaronachermannaaronachermann

instrumenting-with-mlflow-tracing

Instruments Python and TypeScript apps with MLflow tracing for LLM calls and retrievals.

Community
Intermediate
dhar174dhar174
1

langsmith-evaluator

Create and run LangSmith evaluation pipelines for AI agents.

Community
Intermediate