eval-harness

Regression tests for Claude Code tasks are set up using eval-driven development workflows. You define an eval, implement the checks, and run the evaluation to guard against unintended changes across model versions.

Eval-driven development is a formal framework to define, run, and track evaluations for LLM tasks. It measures performance using metrics like pass@k and pass^k to quantify reliability and stability across model versions.

Yes, task evaluation supports deterministic code graders, model-based graders, and human graders. This combination allows you to balance objective determinism with judgment-based reviews within a structured milestone workflow.

Capability tests validate new features by defining an eval, implementing the checks, and executing the workflow. You run the command to execute and review results to ship improvements with confidence.

Metrics like pass@k and pass^k quantify reliability and stability in LLM evaluation. These metrics are applied across capability and regression tests to measure performance and track improvements objectively.