What problem does it solve? AI-assisted coding lacks objective quality gates, making it hard to know whether an agent's output actually works or whether a prompt change broke existing behavior. This Skill provides a formal evaluation framework that treats evals as unit tests for AI development, so you can define pass/fail criteria before coding and measure reliability with pass@k metrics. ## Core Features & Use Cases - Capability and Regression Evals: Define structured eval templates that verify new functionality works and existing behavior stays intact across changes. - Multiple Grader Types: Use deterministic code graders (grep, test runners, builds), model-based LLM-as-judge rubrics, or human review flags for security-sensitive changes. - Reliability Metrics: Track pass@1, pass@3, and pass^3 metrics with recommended thresholds (pass@3 >= 0.90 for capability, pass^3 = 1.00 for release-critical paths). - Use Case: Before adding authentication to your app, define capability evals (user registration, login, session persistence) and regression evals (public routes still work), then run evals after implementation and generate a report showing 5/5 capability and 3/3 regression passes. ## Quick Start Ask the agent to define an eval for a feature you are about to build, including capability evals, regression evals, and pass@k success thresholds stored under .claude/evals/.
