eval-harness

To set up automated evaluation criteria for Claude Code tasks, create an eval definition file under the .claude/evals directory and execute the evaluation using the /eval check command. This formalizes subjective expectations into measurable criteria.

A regression evaluation framework for AI coding agents benchmarks agent behavior across model versions by storing eval definitions and run history. It tracks performance changes over time to prevent capability degradation and ensure consistent code generation quality.

Yes, you can use both deterministic code-based graders and model-based graders for testing Claude Code workflows, plus incorporate human review. This multi-grader approach ensures comprehensive evaluation of agent behavior and task completion accuracy.

To run capability evaluations and benchmark Claude Code sessions, define your feature criteria in the .claude/evals directory, then trigger the evaluation process using /eval check feature-name. The framework generates structured evaluation reports for analysis.

The best way to track evaluation results across different model versions is using a formal eval framework that stores run history and eval definitions. This enables regression tracking, baseline comparisons, and structured reporting for consistent capability assessment.