eval-harness

Regression evals for Claude Code sessions are configured by defining pass/fail criteria and storing definitions in .claude/evals. This framework continuously validates agent behavior and generates reproducible baseline reports.

Eval-driven development defines pass/fail criteria before coding to continuously validate AI agent behavior. It ensures reliability across model versions by supporting capability evals, regression tests, and pass@k metrics.

Yes, evaluating Claude Code tasks supports multiple grader types including model-based, code-based, and human evaluation. These graders integrate into a standardized workflow to verify task success metrics.

Pass@k metrics are tracked by running standardized evaluation workflows stored under .claude/evals. The framework computes these metrics against baselines to measure prompt engineering improvements and regression.

Yes, the framework supports automated grading for model development through code-based and model-based graders. It applies standardized evaluation workflows to ensure reliability across different features and model versions.

A formal evaluation framework is needed when ensuring reliability across model versions, prompt engineering changes, or regression testing. It provides standardized workflows for defining, executing, and tracking capability and regression evals.