eval-harness

Define and run capability and regression evals for AI-assisted development sessions.

Updated Jan 30, 2026

Search Tags:#testing #quality assurance #benchmarking #evaluation #regression #edd

Ready to run

npx skills add https://github.com/ThejanaJayalath/Niolla-PM-system --skill eval-harness-thejanajayalath

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/ThejanaJayalath/Niolla-PM-system/tree/main/.cursor/.agents/skills/eval-harness
Command: npx skills add https://github.com/ThejanaJayalath/Niolla-PM-system --skill eval-harness-thejanajayalath

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill addresses the lack of consistency in AI-assisted development by providing a structured framework to define, track, and verify agent performance before and during implementation.

Core Features & Use Cases

Eval-Driven Development (EDD): Define success criteria and regression tests before writing code to ensure high-quality, reliable outputs.
Multi-Modal Graders: Utilize code-based, model-based, or human-in-the-loop grading to validate task completion and agent reliability.
Performance Metrics: Track pass@k and pass^k metrics to monitor agent reliability trends across model versions and code changes.

Quick Start

Use the eval-harness skill to define a new capability evaluation for the authentication module.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is eval-driven development for AI-assisted coding?▼

Eval-driven development defines success criteria and regression tests before writing code to ensure AI agent outputs are reliable and meet high standards.

How do I set up regression testing for AI agent code generation?▼

You can set up regression testing by defining capability evaluations and using multi-modal graders to verify task completion and track agent reliability across code changes.

Can I use automated grading to evaluate AI agent performance?▼

Yes, automated grading is supported through code-based, model-based, or human-in-the-loop validation to accurately assess task completion and agent reliability.

What metrics should I track to measure AI code generation reliability?▼

Track pass@k and pass^k metrics to monitor agent reliability trends across different model versions and ongoing code modifications during the development lifecycle.

Does this evaluation framework support human-in-the-loop grading?▼

Yes, human-in-the-loop grading is fully supported alongside code-based and model-based analysis to validate task completion and evaluate agent output quality.

When do I need a formal evaluation framework for AI development?▼

You need a formal evaluation framework when consistency in AI-assisted development is lacking, requiring structured tracking to verify agent performance before implementation.