eval-harness

You can run regression evaluations to capture baseline behavior and verify that changes across model versions do not break existing functionality. The harness defines clear pass/fail criteria to track these regressions automatically.

Eval-driven development is a formal evaluation framework for AI coding sessions. It validates model behavior by defining specific tasks, success criteria, and expected outcomes to ensure reliable benchmarks.

You define capability evals by specifying tasks, success criteria, and expected outcomes. The harness supports code-based, model-based, and human graders with pass@k and pass^k metrics to measure results.

Yes, automated grading is supported through code-based and model-based graders. These options evaluate code generation tasks against defined success criteria to produce reliable benchmark metrics.

The best way is using a formal eval harness that automates evaluation across task types. It provides an end-to-end workflow from defining evals to implementing, evaluating, and reporting benchmark results.

No external dependencies are required to set up the AI evaluation workflow. The harness provides a self-contained, end-to-end process for defining, implementing, evaluating, and reporting session results.