eval-harness

Eval-driven development for Claude Code uses a formal framework to define capability and regression evals, run automated grading, and generate measurable quality metrics. This creates repeatable testing throughout the development lifecycle.

Regression evals validate existing behavior by applying code-based, model-based, or human graders to Claude Code sessions. You define checks to protect current functionality and generate reports from centralized run history.

Model-based graders handle open-ended assessment of Claude Code sessions, complementing deterministic code-based checks and manual human review. This grading strategy ensures measurable quality metrics across various eval types.

Pass@k metrics are calculated during the eval report phase, summarizing results from capability and regression runs. The framework coordinates end-to-end execution from definition to centralized result storage for analysis.

The eval framework provides centralized storage for both eval definitions and run history. This ensures rigorous, repeatable testing by maintaining a persistent record of all capability and regression checks.

Automated grading via code-based and model-based graders covers deterministic and open-ended checks, but human graders remain necessary for manual review. Complex edge cases may require manual intervention outside the automated workflow.