langsmith-evaluator

You can evaluate AI agent performance using LangSmith by creating evaluation pipelines that capture agent outputs and trajectories. This involves defining LLM-as-Judge or custom code evaluators and running them locally via the evaluate() function.

To create custom code evaluators for LangChain agent testing, define specific evaluation criteria in Python or TypeScript scripts. These evaluators check agent run outputs against expected formats or outcomes, which can then be uploaded to LangSmith via CLI integration.

LLM-as-Judge evaluators use a language model to grade agent responses against expected outcomes, while custom code evaluators programmatically check for adherence to specific response formats or logic. Both can be defined and executed within LangSmith evaluation pipelines.

Yes, you can run LangSmith evaluations locally without uploading to the cloud. The Skill supports local execution using the evaluate() function, allowing you to test agent outputs and trajectories directly in your Python or TypeScript environment.

To upload Python evaluator scripts to a LangSmith project, use the provided CLI integration. This allows you to automatically execute evaluations by uploading scripts containing your custom evaluator functions directly to your specified LangSmith project.

Yes, this LangSmith evaluation Skill supports both TypeScript and Python. It provides guidance for defining run functions, capturing agent trajectories, and setting up evaluators in either language environment.