eval-harness

To track regressions in AI-generated code, you can define capability and regression evaluations with clear success criteria before coding. Continuously running these evaluations catches regressions introduced by subsequent changes to the agent's logic.

Eval-Driven Development (EDD) is a formal evaluation framework for AI code sessions that uses predefined evaluations to test agent performance. It quantifies reliability using metrics like pass@k and pass^k to ensure AI meets requirements.

You can set up automated grading for AI code evaluations by selecting from code-based, model-based, or human graders. This supports diverse evaluation needs by allowing flexible grading mechanisms tailored to your specific capability tests.

Pass@k and pass^k metrics measure AI agent reliability in code generation. These metrics track whether the agent successfully passes evaluations within a set number of attempts, quantifying both performance and consistency.

Yes, you can define expected behavior and success metrics for a new AI feature before coding by creating capability evaluations. This ensures the AI meets your defined requirements from the start and prevents future regressions.

Model-based graders for AI code evaluation depend on another AI model to judge performance, which may introduce subjective grading biases compared to deterministic code-based graders or precise human grading.