eval-harness

Evaluation-Driven Development (EDD) is a formal framework for evaluating AI coding sessions, ensuring AI reliability by defining expected behavior and success criteria before implementation to track progress.

You can measure AI capability and regression by running continuous evaluations during development, using metrics like pass@k and pass^k to verify features work and catch regressions in existing flows.

To write evaluations before implementation, define capability and regression evaluations to clarify success criteria, then implement the code and run the evaluations to confirm the feature behaves as expected.

Yes, the formal evaluation framework supports integrating both code-based and model-based graders to facilitate structured evaluation workflows from definition to reporting for AI development.

Pass@k and pass^k metrics are used in session evaluation for measuring success rates and reliability, providing quantitative data on AI capability and regression behavior during development.