eval-harness

Eval-driven development (EDD) is a workflow approach that defines and runs formal evaluations to ensure AI tasks meet predefined criteria, measuring reliability through capability and regression tests before and during implementation.

You measure AI reliability by running capability and regression evaluations using code-based, model-based, or human graders, then calculating pass@k metrics to quantify how consistently the AI meets predefined criteria.

You can use three types of graders for AI evaluation: code-based graders for automated checks, model-based graders for AI-assisted scoring, and human graders for manual review of complex outputs.

You should run capability and regression evaluations at three stages: pre-implementation to baseline requirements, during implementation to guide development, and post-implementation to verify AI functionality and store results for review.

Yes, eval-driven development can work without external dependencies, as the framework relies on internal scripts and references to define, run, and store evaluations in a structured format for review.

Pass@k metrics measure AI reliability by checking if a correct response occurs within k attempts, but they do not evaluate the quality of incorrect responses or capture nuanced grading without model-based or human graders.