eval-harness

Eval-driven development uses formal evaluation frameworks to guide AI-assisted software development by defining capability and regression evals with structured success criteria, baselines, and pass/fail reporting to ensure model reliability.

You set up regression evals by defining structured criteria and baselines, then run automated check workflows to measure pass@k metrics across model versions, enabling repeatable and auditable comparisons to detect regressions.

Yes, the evaluation framework supports multi-grader workflows allowing you to combine code-based, model-based, and human review graders to assess AI coding session outputs comprehensively.

You define evals using the define command, execute validation checks with the check command, and generate comprehensive pass/fail evaluation reports using the report command for end-to-end workflow automation.

No external dependencies are required as the framework provides centralized eval storage natively, storing all eval definitions locally under the .claude/evals directory for auditable access.

The best way to automate regression detection is implementing formal evaluation workflows that apply structured criteria and pass@k metrics to generate repeatable, auditable comparisons across different model versions.