eval-harness

Evaluation-driven development for Claude Code tasks is formalized by defining capability and regression evals, implementing deterministic or model-based graders, and running a repeatable workflow to generate auditable reports.

A model-based grader uses an AI model to evaluate task outputs against defined success criteria, complementing deterministic code-based graders to enforce versioned baselines and automate regression testing across multiple model versions.

Regression evals across multiple model versions are executed by defining versioned baselines, applying deterministic or model-based graders, and storing results centrally for human review and audit reporting.

Deterministic code-based graders can be used alongside human review options within the evaluation workflow, enforcing defined success criteria while allowing manual oversight for capability and regression evals.

Versioned baselines for AI capability evaluations are maintained through a formal eval framework that stores organized test definitions and grading results centrally, ensuring repeatable and auditable reporting across development cycles.

Automated model grading should be supplemented with human review when evaluating complex or subjective Claude Code tasks, ensuring defined success criteria are met beyond what deterministic or model-based graders can autonomously verify.