eval-harness

You can define and run formal evaluations for AI-assisted development tasks by applying this framework to Claude Code sessions, executing code- and model-based graders, and recording run history under .claude/evals.

Pass@k reliability measurement tracks the probability that a model's output meets success criteria within k attempts. This framework calculates and reports pass@k and pass^k metrics to benchmark AI capabilities across model versions.

You define success criteria by creating eval definitions using the command to generate a new file in .claude/evals. This allows you to author capability and regression tests that establish measurable thresholds before implementation begins.

Yes, you can use model-based graders to evaluate open-ended outputs. The framework supports deterministic code graders, model-based graders, and human review flags for high-risk changes to ensure comprehensive capability testing.

Yes, eval-driven development works without external dependencies. The framework operates independently to produce structured eval definitions, execute graders, and generate EVAL REPORTs for benchmarking and release gating.

The best way to benchmark AI capabilities is to run formal evaluations that track pass@k and pass^k metrics. This framework stores run history and generates EVAL REPORTs to provide structured benchmarking and release gating across versions.