eval-harness

Set up an eval-driven workflow by defining capability and regression evals, implementing pass/fail criteria, running the eval suite, and reviewing results through a four-phase workflow: Define, Implement, Evaluate, Report.

Capability evals define success criteria to assess AI behavior, while regression evals verify that behavior remains consistent across model iterations. Both are needed to track quality and reliability over time.

Pass@k and pass^k metrics measure model reliability by evaluating successful task completion rates across multiple attempts. These metrics track quality and benchmark performance across different model versions.

The framework supports three grader types: Code-Based Graders for deterministic checks, Model-Based Graders for automated evaluative scoring, and Human Graders for manual review paths.

Yes, you can use this framework to establish regression tests for Claude Code tasks, measure reliability across model versions, and maintain a living eval suite for continuous quality tracking.

This framework requires predefined functional requirements and success criteria before execution. It is structured for rigorous assessment, meaning ad-hoc or undefined Claude Code tasks cannot be reliably evaluated without prior setup.