eval-harness

Name: eval-harness
Availability: InStock
Author: khunglong-03

Automates evaluation of Claude Code tasks with configurable metrics and workflows.

Updated Apr 6, 2026

Search Tags:#evaluation #quality-assurance #regression-testing #eval #ai-evaluation #grader #pass@k

Ready to run

npx skills add https://github.com/khunglong-03/Pathora_Version3 --skill eval-harness-khunglong-03

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/khunglong-03/Pathora_Version3/tree/main/.cursor/skills/eval-harness
Command: npx skills add https://github.com/khunglong-03/Pathora_Version3 --skill eval-harness-khunglong-03

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Eval Harness provides a formal framework to define, run, and track evals for Claude Code sessions, turning evaluations into a repeatable, auditable process.

Core Features & Use Cases

Capability and regression evals to validate behavior before and after changes.
Multiple grader types (Code-Based, Model-Based, Human) to balance determinism with human judgment.
Metrics and workflow support to plan, execute, and report results across model versions and tasks.

Quick Start

Define an eval, implement tests for the feature, run the eval workflow, and generate a report.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up eval-driven development pipelines for Claude Code tasks?▼

To set up eval-driven development pipelines, define an eval, implement tests for the feature, run the structured workflow, and generate a report. This automates rigorous evaluation to ensure task reliability.

What is the difference between capability and regression evals in AI evaluation?▼

Capability evals validate behavior before changes, while regression evals validate behavior after changes. Both are supported to ensure consistent model performance across experiments and versions.

How do pass@k metrics work when benchmarking model versions?▼

The framework supports metrics like pass@k and pass^k to benchmark model versions across experiments. These metrics measure task success rates within the structured workflow from define to report.

Can I use a model-based grader for automated regression testing?▼

Yes, you can use model-based graders for automated regression testing. The framework supports code-based, model-based, and human graders to balance determinism with human judgment during evaluation.