langsmith-evaluator

You can build an LLM evaluation pipeline by creating custom evaluators that inspect agent trajectories and validate tool-use patterns against expected datasets using LangSmith. The pipeline supports both deterministic code checks and LLM-as-Judge logic for subjective quality assessment.

LLM-as-Judge evaluation is a technique using a language model to subjectively assess agent outputs when deterministic code checks are insufficient. You need it for quality assurance when measuring complex response accuracy or validating nuanced agent behavior that static rules cannot capture.

You can run both offline dataset-based evaluations and online project-based quality monitoring within LangSmith. This dual approach allows you to validate expected patterns statically while continuously tracking live agent performance metrics.

You need the LangSmith CLI and specific Python dependencies like langchain-openai and python-dotenv to set up agent evaluators. These integrations capture and validate agent trajectories, enabling the evaluation pipeline to access necessary API configurations.

The best way to define a custom accuracy evaluator is to inspect your agent's output structure and implement tailored evaluation logic within LangSmith. You can apply either deterministic code checks or LLM-as-Judge rules to automatically validate response accuracy.

An LLM evaluation pipeline may fail to capture complex tool-use trajectories if custom evaluators are not properly mapped to the expected output structure. Defining structured validation logic within LangSmith ensures both tool-use patterns and final responses meet accuracy standards.