evals

AI agent evaluation uses three grader types: code-based deterministic checks, model-based nuanced assessment, and human gold standard verification. It analyzes agent transcripts and tool-call sequences to measure capability and regression with pass@k scoring.

Running regression tests for AI agents is best handled through capability evaluations with specific pass targets. This framework automates verification using THE ALGORITHM ISC rows and supports pre-configured domain patterns for coding, conversational, research, and computer-use agents.

Yes, this AI evaluation framework is optimized for Anthropic's Evals Protocol. It integrates with @ai-sdk/anthropic and @langwatch/scenario to run multi-turn scenarios and automated verification.

Yes, model-based grading is supported for nuanced prompt engineering evaluation. It complements code-based deterministic checks and human gold standard grading to assess complex multi-turn agent behaviors.

To run AI agent evaluation scenarios, you need @ai-sdk/anthropic, @langwatch/scenario, and the ai package. These dependencies enable multi-turn scenario execution and model-based grading.

Code-based graders provide deterministic, automated verification for agent testing, while model-based graders handle nuanced, subjective assessments. Combining both with human gold standard checks ensures comprehensive evaluation across multi-turn conversations.