eval-harness

You can set up regression tests for Claude Code by defining evaluation definitions under a versioned eval storage path like .claude/evals, then running them using built-in deterministic or model-based graders to validate behavior across development cycles.

Eval-driven development uses pass@k and pass^k metrics to quantify AI reliability across iterations. It provides a formal evaluation framework that defines baselines and repeatable workflows to track capability and regression evals over time.

Yes, you can use model-based graders alongside deterministic code-based graders for QA automation. This flexible scoring mechanism allows you to evaluate Claude Code sessions using both approaches within the same evaluation framework.

Yes, the eval harness supports pass@k and pass^k metrics specifically for capability evals. These metrics quantify reliability across iterations, helping you track improvements and validate behavior through standardized eval reports.

The best way to define evaluation scope and goals for AI evals is to establish them upfront, then implement eval definitions under a versioned storage path. This approach enables repeatable assessments using workflow hooks and standardized reporting.

Use capability evals to validate new behaviors and regression evals to ensure existing functionality remains intact. Both evaluation types integrate with versioned eval storage and workflow hooks to guide improvements through standardized reports.