eval-harness

Regression testing for AI coding sessions involves defining explicit pass/fail success criteria for existing functionality before code changes. You can run evals during implementation to catch regressions and confirm features work without breaking current flows before merging.

Eval-driven development is a formal framework that defines explicit pass/fail criteria for AI-assisted coding tasks. It eliminates guesswork by measuring agent reliability and catching regressions early, ensuring consistent and predictable outcomes from AI coding sessions.

Measuring agent reliability with pass@k metrics involves running structured eval workflows that track success rates across multiple attempts. The framework supports pass@k and pass^k reliability metrics to benchmark agent performance across model versions.

Yes, you can use deterministic code-based graders for automated checks on AI-generated code. For open-ended output evaluation, model-based graders are available, and high-risk modifications can be flagged for human review.

Yes, the eval framework works without external dependencies. It stores eval artifacts in project directories for version control, allowing you to define and run evals within your existing development environment.

Use deterministic code-based graders for automated checks and model-based graders for open-ended output evaluation. Flag changes for human review when dealing with high-risk modifications that require manual verification before merging.