eval-harness

Define capability and regression evals with explicit success criteria before writing any code, then implement the feature and run the evals to verify. Store eval definitions in a project directory like .claude/evals/ so they are versioned alongside the code.

pass@k measures whether at least one of k attempts succeeds, useful for general reliability targets like pass@3 above 90%. pass^k requires all k trials to succeed consecutively, a stricter bar suited for critical regression paths.

Record a baseline checkpoint such as a commit SHA, list the existing behaviors that must keep passing, and run deterministic checks like npm test or build commands after each change. Report results as passed count versus the baseline.

Use code-based graders for deterministic checks like pattern matching, test suites, and build success since they are reproducible. Use model-based graders only for open-ended qualities like code structure and edge-case handling, and flag security-sensitive changes for human review.

Automated evals cannot fully cover security review, which the framework explicitly reserves for human graders. Slow evals also tend to be skipped, so checks must stay fast, and evals only measure the criteria you defined upfront.