What problem does it solve?
When building AI-assisted workflows with Claude Code, teams lack a standardized way to define clear success criteria, measure agent reliability, catch regressions from code or prompt changes, and track performance across model versions, leading to unvalidated, unreliable outputs that break unexpectedly.
Core Features & Use Cases
- Standardized Eval Definitions: Create structured capability and regression eval documents with clear pass/fail criteria before starting development work.
- Flexible Grader System: Use code-based graders for deterministic checks, model-based graders for open-ended output evaluation, or flag items for human review for high-risk changes.
- Reliability Metrics: Track pass@k (success within k attempts) and pass^k (all k consecutive successes) metrics to measure and improve agent consistency over time.
- Use Case: A developer adding a new user authentication feature can use this skill to define evals for the new capability, run regression evals to ensure existing login flows still work, and generate a report to confirm the feature meets reliability thresholds before shipping.
Quick Start
Ask Claude to use the eval-harness skill to define capability and regression evals for your new Claude Code feature and generate a baseline performance report.
