Implement eval-driven development with code, model, and human graders.
npx skills add https://github.com/YosefHayim/Template --skill eval-harness-yosefhayim
Please help me install this Agent Skill. Skill: eval-harness Source: https://github.com/YosefHayim/Template/tree/main/.cursor/skills/eval-harness Command: npx skills add https://github.com/YosefHayim/Template --skill eval-harness-yosefhayim
This Skill provides a structured approach to evaluating AI-generated code and agent performance, ensuring reliability and preventing regressions.
Define a new capability evaluation for a feature named 'user-authentication'.
High-intent search queries and answers about installing and using this skill.
To implement eval-driven development, define capability and regression success criteria before implementation, then run code-based, model-based, and human graders to measure AI code reliability using pass@k metrics.
Pass@k metrics measure AI code reliability by calculating the probability that at least one generated code sample passes the defined evaluations. The framework supports these metrics alongside pass^k to track consistency across AI agent capabilities.
Set up regression testing for AI-generated code by defining regression evaluations and storing them in the project's .claude/evals directory. Run these evals after implementation to confirm new features haven't broken existing functionality.
Yes, you can evaluate AI agent performance using code-based, model-based, and human graders. This allows flexible assessment of AI outputs depending on the complexity of the capability being tested and the required evaluation strictness.
No, the evaluation framework operates without external dependencies. It manages evaluation storage and configuration natively within a project's .claude/evals directory for continuous integration.
Define capability evaluations before implementation begins. This ensures AI development is guided by formal success criteria, allowing you to run the evaluations afterward to verify the feature works and hasn't introduced regressions.
jinyuanluDefine and run eval-driven development tests for Claude Code sessions.
daeha76Create and execute capability and regression evaluations with code, model, or human grading.
JamkrisAutomate eval-driven testing workflows for Gemini CLI sessions with deterministic graders.
rivaldiekaptrrrDefine and run eval suites to validate AI task performance.
