eval-harness

Eval-driven development systematically evaluates Claude Code sessions by defining capability and regression checks, applying graders, and tracking metrics like pass@k to quantify reliability and detect regressions.

Set up regression evals by defining expected behavior checks, configuring code-based or model-based graders, and storing eval definitions and baselines as artifacts to capture run histories for traceability.

Yes, the framework supports code-based, model-based, and human graders alongside pass@k and pass^k metrics to quantify both the reliability of capability evaluations and the presence of regressions.

Track regressions by storing eval definitions, baselines, and run histories as artifacts, using pass^k metrics to monitor reliability across capability and regression eval runs for reproducible reporting.

No external dependencies are required to run capability evaluations; the framework specifies requirements for eval definitions, graders, metrics, and artifacts directly within Claude Code sessions.

A formal evaluation framework is unnecessary for simple, one-off Claude Code tasks where systematic defect-detection, regression tracking, and reproducible reporting are not required.