eval-harness

Eval-driven development treats AI evals as unit tests for AI development, requiring success criteria to be defined before implementation begins. This approach ensures measurable validation of Claude Code agent performance and consistent reliability across prompt changes.

You can set up a regression test suite by defining capability and regression evals with specific reliability targets like pass@3. This Skill supports deterministic code-based graders, LLM-as-judge graders, and human review flags to evaluate code generation consistency.

Yes, agent evaluation supports multiple configurable grader types including deterministic code-based graders, LLM-as-judge model graders, and human review flags. This allows you to handle different levels of evaluation complexity within the same framework.

Yes, benchmarking agent performance across model versions requires structured pass/fail evaluation criteria defined before implementation. This Skill enforces these criteria to track pass@k and pass^k metrics, measuring agent consistency and catching regressions.

Yes, benchmarking agent performance across model versions requires structured pass/fail evaluation criteria defined before implementation. This Skill enforces these criteria to track pass@k and pass^k metrics, measuring agent consistency and catching regressions.