eval-harness

Define and run pass/fail evaluations with regression testing for AI agents.

1|Updated Apr 11, 2026

Search Tags:#testing #qa #benchmarking #reliability #evals #edd

Ready to run

npx skills add https://github.com/its-Basudeba/Care-HMS --skill eval-harness-its-basudeba

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/its-Basudeba/Care-HMS/tree/main/.agent/.agents/skills/eval-harness
Command: npx skills add https://github.com/its-Basudeba/Care-HMS --skill eval-harness-its-basudeba

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill addresses the lack of formal testing in AI-assisted development by providing a structured framework to define, run, and report on agent performance using Eval-Driven Development (EDD) principles.

Core Features & Use Cases

Capability Evals: Define and verify new agent features with clear success criteria.
Regression Testing: Ensure new changes do not break existing functionality using baseline comparisons.
Multi-Modal Grading: Support for deterministic code-based checks, model-based evaluation of open-ended outputs, and human-in-the-loop review.
Use Case: Use this to benchmark a new coding agent's performance across different model versions or to ensure that a complex refactor maintains existing system reliability.

Quick Start

Use the eval-harness skill to define a new capability evaluation for the current feature set.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I implement eval-driven development for AI agents?▼

Eval-driven development for AI agents is implemented by defining pass/fail criteria and regression testing workflows to measure agent reliability using structured evaluation frameworks and pass@k metrics.

What is the best way to benchmark AI agent reliability across model versions?▼

Benchmarking AI agent reliability involves running structured capability evaluations with deterministic code-based graders and model-based assessments to compare performance across different model versions.

How do I set up regression testing for complex agent refactors?▼

Regression testing for agent refactors uses baseline comparisons and formal evaluation frameworks to ensure new changes do not break existing functionality or degrade agent performance.

Can I use model-based evaluation for open-ended agent outputs?▼

Model-based evaluation supports complex task validation for open-ended outputs, complementing deterministic code-based checks and human-in-the-loop review within a multi-modal grading framework.

Does eval-driven development require predefined pass/fail criteria for capability evaluations?▼

Capability evaluations require clear success criteria defined as pass/fail conditions to formally verify new agent features and measure reliability through structured pass@k metrics.

When should I use human-in-the-loop review instead of deterministic code-based graders?▼

Human-in-the-loop review is utilized for complex task validation when deterministic code-based graders are insufficient, enabling multi-modal grading within a structured evaluation workflow.