eval-harness

An AI agent evaluation framework defines pass/fail criteria to measure agent performance. You need it to validate new capabilities, prevent regressions, and ensure reliable behavior through structured evals before and after coding.

To set up regression and capability evals, define pass/fail criteria using /eval define, implement evaluators, run /eval check to validate behavior, and generate a report with /eval report for traceable performance metrics.

Available grader types for evaluating AI agent tasks include Code-Based graders for deterministic checks, Model-Based graders for qualitative assessments, and Human graders to cover mixed evaluation requirements.

Eval-driven development works by defining success criteria before coding, implementing evaluators, running checks with /eval, and storing results under .claude/evals to continuously validate AI agent behavior and ensure measurable improvements.

Yes, you can track pass@k and pass^k metrics. The framework computes these metrics and generates eval reports, providing reproducible performance tracking and auditing for your AI agent evaluations.

AI agent evaluation results, definitions, and histories are stored project-scoped under the .claude/evals directory, ensuring centralized storage for reproducibility, baselines, and auditing across evaluation runs.