eval-harness

Run regression testing for LLM reliability by defining baseline capability checks and tracking X/Y pass results over iterative prompt changes. This eval-driven workflow records PASS/FAIL outcomes to prevent regressions.

Eval-driven development measures prompt quality by applying explicit success criteria to AI outputs. It uses deterministic code-based graders, rubric-based model graders, and human review gates to quantify agent reliability and pass@k metrics.

Define capability evals by creating a YAML-identified skill entry with explicit success criteria for what Claude must achieve. Run the checks and record the resulting PASS/FAIL outcomes to verify newly implemented capabilities.

You can evaluate LLM outputs using multi-mode grading: deterministic code-based graders for strict checks, rubric-based model graders for subjective quality, and human review gates for high-risk or ambiguous outputs.

Track pass@k reliability by running repeatable evaluation checks and generating eval reports that record X/Y pass results over time. This workflow measures stability and prevents regressions across iterative changes.

No external dependencies are required to set up an eval harness for Claude Code. The workflow relies on a YAML-identified skill entry plus instruction-driven patterns for eval definitions, grader types, metrics, and storage layout.