eval-harness

Defines and runs pass/fail evaluations for AI-assisted coding tasks.

1|Updated Apr 6, 2026

Search Tags:#testing #benchmarking #evaluation #reliability #regression #edd

Ready to run

npx skills add https://github.com/vrcms/everything-qwen-code --skill eval-harness-vrcms

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/vrcms/everything-qwen-code/tree/main/.qwen/skills/eval-harness
Command: npx skills add https://github.com/vrcms/everything-qwen-code --skill eval-harness-vrcms

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill addresses the lack of reliability in AI-assisted coding by providing a structured framework to define, run, and report on evaluation criteria before and during development.

Core Features & Use Cases

Eval-Driven Development (EDD): Establish pass/fail criteria for tasks to ensure consistent agent performance.
Multi-Modal Grading: Supports deterministic code-based checks, model-based evaluation for open-ended outputs, and human-in-the-loop review.
Reliability Metrics: Tracks pass@k and pass^k metrics to measure agent stability and regression resistance over time.

Quick Start

Use the eval-harness skill to define a new evaluation suite for the current feature by running the command to initialize the evaluation definition file.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is eval-driven development for AI coding agents?▼

Eval-driven development establishes formal pass/fail criteria before and during AI-assisted coding sessions to ensure consistent agent performance and reliability. It provides a structured framework to define, run, and report on evaluation criteria.

How do I set up regression testing for AI-assisted coding tasks?▼

Set up regression testing by initializing an evaluation definition file to establish pass/fail criteria. The framework tracks pass@k and pass^k metrics to measure agent stability and regression resistance over time.

Can I use model-based evaluation for open-ended AI agent outputs?▼

Yes, model-based evaluation is supported for open-ended outputs. The framework facilitates multi-modal grading, including deterministic code-based checks, model-based evaluation, and human-in-the-loop adjudicated review.

What's the best way to benchmark AI agent reliability during development?▼

Benchmark AI agent reliability by tracking pass@k and pass^k metrics within a formal evaluation framework. This measures agent stability and regression resistance to ensure high-quality output during development.

When do I need human-in-the-loop review for AI coding evaluation?▼

Human-in-the-loop review is needed for complex evaluation criteria requiring human adjudication. It complements deterministic code-based checks and model-based evaluation to ensure high-quality output and stability.