skills.rest
langchain-ai
langchain-aiOFFICIAL
Explore all skills by this author →
Website𝕏@LangChainGitHub

langsmith-online-eval-engineering

Creates LangSmith online evaluators by inspecting traces and iteratively building LLM-as-judge or code evaluators.

1.2k|90|Updated Jan 22, 2026
Search Tags:#LLM-as-judge#LangSmith#trace inspection#Observability#Evaluator#online evaluation#run rules
One-click install
npx skills add https://github.com/langchain-ai/langchain-skills --skill langsmith-online-eval-engineering
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: langsmith-online-eval-engineering
Source: https://github.com/langchain-ai/langchain-skills/tree/main/config/skills/langsmith-online-eval-engineering
Command: npx skills add https://github.com/langchain-ai/langchain-skills --skill langsmith-online-eval-engineering

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires langsmith, httpx, pydantic, langchain-core, and includes references (resource) components.

What problem does it solve?

Building effective online evaluators for LangSmith requires understanding actual trace structures, choosing the right evaluation approach, and correctly wiring evaluators to projects via run rules. This Skill guides an iterative workflow that inspects real traces, interviews the user about quality concerns, and creates verified evaluators one at a time.

Core Features & Use Cases

  • Trace Inspection: Fetches recent root-level traces from a LangSmith project to discover input/output field names and data shapes before building anything.
  • Evaluator Design Guidance: Proposes two or three evaluation criteria grounded in trace data, choosing between LLM-as-judge and code evaluators based on whether the criterion is semantic or deterministic.
  • End-to-End Evaluator Creation: Builds the evaluator (structured prompt with reasoning-first schema, or self-contained Python perform_eval function), tests it against historical traces, and attaches it via run rules with a user-chosen sampling rate.
  • Use Case: A team running a RAG chatbot wants to monitor response relevance in production. The Skill inspects their traces, proposes a relevance LLM-as-judge evaluator, creates it in LangSmith, tests it on existing traces, and attaches it with a 1.0 sampling rate.

Quick Start

Ask the agent to inspect traces from your LangSmith project and help you create an online evaluator for a quality concern you care about.

Frequently Asked Questions about langsmith-online-eval-engineering

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create an online evaluator in LangSmith?▼

Inspect recent traces from your LangSmith project to discover field names, then create either an LLM-as-judge evaluator (structured prompt pushed to the Prompt Hub) or a code evaluator (a perform_eval function). Attach it to your project with a run rule specifying a sampling rate.

LLM-as-judge vs code evaluator: which should I use?▼

Use a code evaluator when pass/fail is decidable by a Python expression, such as format validation or length thresholds. Use an LLM-as-judge when the criterion is semantic, like relevance or tone, and requires reading comprehension.

Why does my LangSmith code evaluator fail with AttributeError?▼

The run parameter is a plain dict at runtime, not an object. Use run.get("inputs") and run.get("outputs") instead of attribute access, and define perform_eval(run, example=None) since the runtime calls it with one argument.

How do I test an online evaluator before attaching it?▼

Run rules only fire on new traces, so test against historical traces first. For code evaluators, execute perform_eval directly against fetched root-level traces; for LLM evaluators, verify variable_mapping keys match prompt placeholders and trace fields.

What sampling rate should I use for LangSmith run rules?▼

A sampling rate of 1.0 evaluates every trace and is recommended for initial testing. Lower rates like 0.5 or 0.1 reduce evaluation volume and cost once the evaluator is verified to work correctly.

Related Skills

Explore More Software Engineering Skills→
infantesromeroadrianinfantesromeroadrian

langsmith

Trace and inspect LLM workflow runs with the LangSmith CLI.

Community
Advanced
tankpkgtankpkg
1

@tank/llm-observability

Diagnose LLM observability and evaluation gaps across production tracing and prompt workflows.

Official
Advanced
samcdavidsamcdavid
4

my-eval-plan

Design evaluation plans for AI/LLM features with scorers, datasets, and baselines.

Community
Advanced
ContextJet-aiContextJet-ai
29

add-llm-evals

Add offline and online evaluation scoring to LLM and agent applications.

Official
Advanced