eval-harness

Implement Eval-Driven Development evaluations with code, model, and human graders.

1|Updated Feb 22, 2026

Search Tags:#code quality #testing #AI development #metrics #evaluation #regression testing

Ready to run

npx skills add https://github.com/TakMczk/copilot-cli-ecc --skill eval-harness-takmczk

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/TakMczk/copilot-cli-ecc/tree/main/.github/skills/eval-harness
Command: npx skills add https://github.com/TakMczk/copilot-cli-ecc --skill eval-harness-takmczk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a structured framework for evaluating AI-generated code and workflows, ensuring reliability and quality through Eval-Driven Development (EDD) principles.

Core Features & Use Cases

Define Success Criteria: Clearly articulate expected outcomes before implementation.
Automated Evaluation: Implement various grader types (code, model, human) to assess AI outputs.
Metrics Tracking: Measure performance using pass@k and pass^k metrics for reliability.
Use Case: When developing a new AI agent for code refactoring, use this Skill to define capability and regression evals, run them against the agent's output, and track its pass@3 rate to ensure consistent quality.

Quick Start

Use the eval-harness skill to define a new capability evaluation for user authentication.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is Eval-Driven Development for AI code generation?▼

You can evaluate AI code outputs using code-based, model-based, and human graders within this evaluation framework. These grader types assess AI-generated code against defined success criteria to measure performance and ensure consistent quality.

How do I track AI code regression testing metrics?▼

You track AI code regression testing metrics using pass@k and pass^k measurements. This evaluation framework calculates these metrics across capability and regression evals to monitor reliability and catch performance degradation in AI-assisted code workflows.

How do I set up a capability evaluation for an AI coding agent?▼

You need to define expected outcomes and implement graders to evaluate AI-generated code. Graders can be code-based, model-based, or human, allowing you to structure AI development and quality assurance without requiring external dependencies.

When should I use formal evaluation metrics for AI-generated code?▼

You should use formal evaluation metrics when developing new AI agents or workflows where reliability is critical. Defining capability and regression evals with pass@k metrics ensures your AI-assisted code development meets consistent quality standards.