eval-harness

Regression testing for Claude Code is handled by defining formal evaluation suites that validate features and detect regressions. You define capability and regression evals before implementation to clarify expected behavior and track success criteria.

Eval-driven development defines capability and regression evals before implementation to clarify expected behavior. It measures AI code reliability using pass@k metrics, applying formal evaluation suites to validate changes and track success criteria over time.

Code evaluation supports multi-modal grading by applying deterministic shell-compatible code-based graders for automated checks, model-based graders for open-ended outputs, and human review flags for high-risk changes to validate AI-driven code.

Yes, evaluation definitions and run history are stored as artifacts under the .claude/evals directory. This allows you to version evals alongside your code and generate per-feature reports for CI and release decisions.

The best way to measure pass@k reliability during CI is to run formal evaluation suites that track pass@k and pass^k metrics over time. The framework generates per-feature eval reports to inform automated release decisions.

Automated grading should be supplemented or avoided for high-risk changes where deterministic checks are insufficient. The framework supports human review flags to manually grade open-ended outputs that model-based or code-based graders cannot reliably evaluate.