What problem does it solve? AI-assisted coding often lacks a systematic way to verify that changes work as intended and do not break existing behavior. This Skill provides a formal evaluation framework that treats evals as unit tests for AI development, letting you define success criteria before coding and measure reliability with pass@k metrics. ## Core Features & Use Cases - Capability and Regression Evals: Define structured eval templates that test new features and guard existing functionality against regressions. - Multiple Grader Types: Use code-based graders (grep, npm test, build checks), model-based graders for open-ended output scoring, and human review flags for high-risk changes. - Reliability Metrics: Track pass@k and pass^k metrics to measure first-try success rates and consistency across repeated trials. - Use Case: When adding authentication to an app, define evals for signup, login, and session persistence before writing code, then run /eval check add-authentication during implementation and generate a final report showing 5/5 capability evals passing. ## Quick Start Ask the AI to define an eval for a new feature using the eval-harness framework, then implement the code and run the eval check to generate a pass/fail report.
