eval-harness

To set up an evaluation framework for Claude Code sessions, define a new eval feature in your project and run the eval workflow to generate a report. This process uses an eval-driven development approach to ensure consistent quality checks across model iterations, prompts, and tooling changes.

Eval-driven development is the process of defining and running formal evaluations to validate model behavior and regression safety. You need it when applying capability and regression scenarios across model iterations, prompt updates, or tooling changes to quantify reliability and stability.

Yes, you can use deterministic code-based graders, model-based graders, and human graders for robust assessment. This allows you to flexibly evaluate different aspects of Claude Code sessions using the most appropriate grading mechanism for each scenario.

You track pass@k and pass^k metrics through a reusable eval workflow and centralized storage. These metrics quantify the reliability and stability of your evaluations across different capability and regression scenarios during model benchmarking.

Relying on a single grader type for regression evals can limit assessment robustness. The framework supports deterministic code-based, model-based, and human graders simultaneously to mitigate blind spots and ensure comprehensive validation of model behavior changes.