eval-harness

Regression tests for Claude Code workflows are set up using a formal evaluation framework that defines capability evaluations, graders, and metrics across model versions. It supports code-based, model-based, and human graders with structured storage.

Eval-driven development for AI coding sessions is a structured testing approach using a formal harness to define benchmarks, apply graders, and measure pass-at-k metrics. It ensures reproducible results and reliable capability tracking across model versions.

Model-based grading is fully supported for evaluating Claude Code capabilities alongside code-based and human graders. The framework allows you to combine multiple grader types to measure metrics and establish baselines for different model versions.

Pass-at-k metrics for code generation models are measured by running the evaluation harness to execute defined capability tests and apply graders. The framework stores results to establish clear baselines and track regression across model versions.

Evaluation data for AI model grading should follow the framework's documented storage and guideline layout. This structured approach ensures reproducible benchmarks and reliable metrics when applying code-based, model-based, or human graders.

Automated graders for regression evaluation work well for code-based and model-based metrics, but human graders are also supported for subjective or complex capability assessments. Combining grader types provides the most reliable results across model versions.