eval-harness

Eval-driven development (EDD) treats AI evaluations as unit tests, defining expected model behavior before implementation begins. This approach ensures structured AI development by establishing success criteria upfront and continuously monitoring for regressions.

You can set up capability evaluations to test new AI functionalities and regression evaluations to ensure changes don't break existing features. The harness supports code-based, model-based, and human grading to measure pass@k and pass^k metrics.

Regression testing tracks AI model reliability using pass@k and pass^k metrics. These metrics measure consistency and reliability, helping teams monitor whether existing capabilities are broken by recent code changes.

Yes, capability evaluations support automated grading through code-based and model-based graders. This allows teams to automatically confirm new AI features meet defined requirements without relying solely on human grading.

Model-based grading automates the evaluation of AI capabilities for faster regression testing, while human grading provides subjective assessment for complex behaviors. The harness supports both, along with code-based grading, for flexible evaluation.

Yes, this formal evaluation framework operates without external dependencies, allowing teams to implement eval-driven development principles and track pass@k metrics directly within their AI development sessions.