eval-harness

Name: eval-harness
Availability: InStock
Author: thanhquan3010

Define capability and regression evaluations with code, model, or human graders.

Updated Mar 7, 2026

Search Tags:#testing #ai development #metrics #evaluation #regression #edd

Ready to run

npx skills add https://github.com/thanhquan3010/hospital-service-management-system --skill eval-harness-thanhquan3010

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/thanhquan3010/hospital-service-management-system/tree/main/.agent/skills/eval-harness
Command: npx skills add https://github.com/thanhquan3010/hospital-service-management-system --skill eval-harness-thanhquan3010

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides a structured framework for evaluating AI-generated code and agent performance, enabling rigorous testing and regression tracking through Eval-Driven Development (EDD).

Core Features & Use Cases

Define Evals: Create capability and regression evaluations with clear success criteria.
Automated Grading: Supports code-based, model-based, and human grading for diverse evaluation needs.
Metrics Tracking: Implements pass@k and pass^k metrics to quantify reliability and performance.
Use Case: When developing a new AI feature, define its expected behavior and success metrics before coding. Continuously run these evaluations to ensure the AI meets the requirements and to catch regressions introduced by subsequent changes.

Quick Start

Use the eval-harness skill to define a new capability evaluation for user authentication.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I track regressions in AI-generated code during development?▼

To track regressions in AI-generated code, you can define capability and regression evaluations with clear success criteria before coding. Continuously running these evaluations catches regressions introduced by subsequent changes to the agent's logic.

What is Eval-Driven Development for AI agent performance?▼

Eval-Driven Development (EDD) is a formal evaluation framework for AI code sessions that uses predefined evaluations to test agent performance. It quantifies reliability using metrics like pass@k and pass^k to ensure AI meets requirements.

How do I set up automated grading for AI code evaluations?▼

You can set up automated grading for AI code evaluations by selecting from code-based, model-based, or human graders. This supports diverse evaluation needs by allowing flexible grading mechanisms tailored to your specific capability tests.

What metrics measure AI agent reliability in code generation?▼

Pass@k and pass^k metrics measure AI agent reliability in code generation. These metrics track whether the agent successfully passes evaluations within a set number of attempts, quantifying both performance and consistency.

Can I define expected behavior for a new AI feature before coding?▼

Yes, you can define expected behavior and success metrics for a new AI feature before coding by creating capability evaluations. This ensures the AI meets your defined requirements from the start and prevents future regressions.

What are the limitations of using model-based graders for AI code evaluation?▼

Model-based graders for AI code evaluation depend on another AI model to judge performance, which may introduce subjective grading biases compared to deterministic code-based graders or precise human grading.