eval-harness

Name: eval-harness
Availability: InStock
Author: yosefhayim

Implement eval-driven development with code, model, and human graders.

Updated Aug 23, 2026

Search Tags:#code quality #testing #ai development #metrics #evaluation #regression testing

Ready to run

npx skills add https://github.com/YosefHayim/Template --skill eval-harness-yosefhayim

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/YosefHayim/Template/tree/main/.cursor/skills/eval-harness
Command: npx skills add https://github.com/YosefHayim/Template --skill eval-harness-yosefhayim

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides a structured approach to evaluating AI-generated code and agent performance, ensuring reliability and preventing regressions.

Core Features & Use Cases

Eval-Driven Development (EDD): Define success criteria before implementation to guide AI development.
Automated Testing: Implement various grader types (code-based, model-based, human) to assess AI outputs.
Metrics Tracking: Measure performance using pass@k and pass^k metrics for reliability.
Use Case: When developing a new feature, define capability and regression evals first. After implementation, run these evals to confirm the feature works as expected and hasn't broken existing functionality.

Quick Start

Define a new capability evaluation for a feature named 'user-authentication'.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I implement eval-driven development for AI code sessions?▼

To implement eval-driven development, define capability and regression success criteria before implementation, then run code-based, model-based, and human graders to measure AI code reliability using pass@k metrics.

What are pass@k metrics for measuring AI code reliability?▼

Pass@k metrics measure AI code reliability by calculating the probability that at least one generated code sample passes the defined evaluations. The framework supports these metrics alongside pass^k to track consistency across AI agent capabilities.

How do I set up regression testing for AI-generated code?▼

Set up regression testing for AI-generated code by defining regression evaluations and storing them in the project's .claude/evals directory. Run these evals after implementation to confirm new features haven't broken existing functionality.

Can I use different grader types to evaluate AI agent performance?▼

Yes, you can evaluate AI agent performance using code-based, model-based, and human graders. This allows flexible assessment of AI outputs depending on the complexity of the capability being tested and the required evaluation strictness.

Does this evaluation framework require external dependencies to run?▼

No, the evaluation framework operates without external dependencies. It manages evaluation storage and configuration natively within a project's .claude/evals directory for continuous integration.

When should I define capability evaluations in the development lifecycle?▼

Define capability evaluations before implementation begins. This ensures AI development is guided by formal success criteria, allowing you to run the evaluations afterward to verify the feature works and hasn't introduced regressions.