eval-harness

To set up an evaluation framework for AI agent performance, define your eval parameters, implement the specific checks using code, model, or human graders, and run the evaluation to generate a detailed performance report.

Eval-driven development is a structured process to measure AI model performance quantitatively. You need it to reduce guesswork and prevent regressions when validating new capabilities or guarding existing functionality across updates.

You use pass@k and pass^k metrics within an evaluation framework to quantify AI reliability by running deterministic checks against your agents. These metrics measure capability and regression reliability during the evaluation workflow.

Yes, you can use human graders alongside code and model graders for AI quality assurance. You align the grader type with your specific risk and context to ensure the evaluation rigor matches the required safety and accuracy constraints.

The best way to run regression tests for Claude Code sessions is using a formal eval framework that applies capability and regression evaluations with deterministic metrics, ensuring existing functionality is guarded across software updates.

To implement capability evals for new AI features, define the expected outcomes, implement the checks using your chosen grader types, and execute the evaluation workflow to generate a report validating the new capability.