eval-harness

Regression testing for Claude Code sessions is handled by defining regression evals that compare current outputs against past runs and baselines. The framework supports automated checks to detect regressions and track success criteria across tasks.

A capability eval defines new abilities for Claude Code before integration. It provides a formal, repeatable framework to measure and track success criteria across tasks to ensure new features function as intended.

Yes, Claude evaluation supports configurable grading that combines deterministic code graders, model graders, and human reviewers. This grader flexibility ensures balanced assessment across different evaluation types.

Pass@k metrics in Claude Code evaluation measure success rates across multiple attempts by applying formal checks against definable eval definitions. The framework automates these checks to track and compare results against baselines.

Automating Claude Code regression checks is best done by defining regression evals within a standardized workflow lifecycle. This enables automated checks from definition to reporting, detecting regressions by comparing to baselines and past runs.

No dependencies are required to run the Claude Code evaluation framework. It operates independently to provide formal eval definitions, automated checks, and configurable grading for deterministic code graders, model graders, and human reviewers.