eval-harness

Eval-Driven Development (EDD) formalizes the evaluation of AI-generated outputs by defining expected behavior before implementation. It uses capability and regression evals with code-based, model-based, and human grading mechanisms to ensure reliable AI-assisted workflows.

You can create regression test suites for AI-generated features by defining success criteria using evaluation templates before implementation. The framework measures agent reliability using pass@k and pass^k metrics to ensure new updates do not break existing functionality.

AI agent reliability is measured using pass@k and pass^k metrics. These metrics track the consistency and reliability of AI-generated outputs across multiple attempts, providing quantitative data for capability evaluation and regression testing.

Yes, the evaluation framework supports code-based, model-based, and human grading mechanisms. These grading methods can be combined within capability and regression evaluations to rigorously assess AI-generated outputs against predefined success criteria.

You define success criteria for a new AI feature by applying Eval-Driven Development principles, which require establishing expected behavior through evaluation templates prior to writing the implementation, ensuring outputs meet measurable benchmarks.

Regression evals should be used when modifying or adding AI features to verify that new implementations meet defined success criteria and do not break existing functionality. They track pass^k metrics to measure ongoing reliability across development iterations.