eval-harness

You can automate regression testing for Claude Code sessions by codifying evaluation definitions and running automated checks to measure reliability with pass@k and pass^k metrics.

Evaluation-Driven Development (EDD) is a process to define, run, and track formal evaluations across features, model prompts, and conversation contexts to measure AI coding reliability.

Measure pass@k and pass^k metrics by running capability and regression assessments using flexible scoring via code-based, model-based, and human graders, generating structured reports with baseline comparisons.

Yes, formal evaluations support easy integration into development workflows and CI-like environments, allowing you to define evaluations, execute automated checks, and generate structured reports.

Automated evaluation checks support flexible scoring via code-based graders, model-based graders, and human graders to accurately assess capabilities across various conversation contexts.

Generate structured reports comparing baselines by executing automated checks on defined evaluations, which outputs structured reporting including baseline comparisons and scorer outputs.