What problem does it solve? AI-assisted coding lacks a systematic way to verify that Claude Code sessions actually accomplish their goals and that changes do not break existing functionality. This Skill provides a formal evaluation framework that treats evals as the unit tests of AI development, letting you define success criteria before implementation and measure agent reliability quantitatively. ## Core Features & Use Cases - Capability and Regression Evals: Define structured eval templates that test whether Claude can perform new tasks and whether changes break existing behavior. - Multiple Grader Types: Use deterministic code-based graders, model-based LLM-as-judge graders, or human review flags depending on the output being evaluated. - Reliability Metrics: Track pass@k (at least one success in k attempts) and pass^k (all k trials succeed) to benchmark agent performance across model versions. - Use Case: Before adding an authentication feature, define capability evals (user registration, login, session persistence) and regression evals (existing routes unchanged), then run evals after implementation and generate a report showing pass@3 of 100% before shipping. ## Quick Start Ask Claude to define an eval for a new feature using the eval-harness framework, including capability evals, regression evals, and pass@k success thresholds stored under .claude/evals/.
