skills.rest
khunglong-03
khunglong-03COMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Automates evaluation of Claude Code tasks with configurable metrics and workflows.

Updated Apr 6, 2026
Search Tags:#evaluation#quality-assurance#regression-testing#eval#ai-evaluation#grader#pass@k
One-click install
npx skills add https://github.com/khunglong-03/Pathora_Version3 --skill eval-harness-khunglong-03
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/khunglong-03/Pathora_Version3/tree/main/.cursor/skills/eval-harness
Command: npx skills add https://github.com/khunglong-03/Pathora_Version3 --skill eval-harness-khunglong-03

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Eval Harness provides a formal framework to define, run, and track evals for Claude Code sessions, turning evaluations into a repeatable, auditable process.

Core Features & Use Cases

  • Capability and regression evals to validate behavior before and after changes.
  • Multiple grader types (Code-Based, Model-Based, Human) to balance determinism with human judgment.
  • Metrics and workflow support to plan, execute, and report results across model versions and tasks.

Quick Start

Define an eval, implement tests for the feature, run the eval workflow, and generate a report.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up eval-driven development pipelines for Claude Code tasks?▼

To set up eval-driven development pipelines, define an eval, implement tests for the feature, run the structured workflow, and generate a report. This automates rigorous evaluation to ensure task reliability.

What is the difference between capability and regression evals in AI evaluation?▼

Capability evals validate behavior before changes, while regression evals validate behavior after changes. Both are supported to ensure consistent model performance across experiments and versions.

How do pass@k metrics work when benchmarking model versions?▼

The framework supports metrics like pass@k and pass^k to benchmark model versions across experiments. These metrics measure task success rates within the structured workflow from define to report.

Can I use a model-based grader for automated regression testing?▼

Yes, you can use model-based graders for automated regression testing. The framework supports code-based, model-based, and human graders to balance determinism with human judgment during evaluation.

Related Skills

Explore More Software Engineering Skills→
jinyuanlujinyuanlu
3

eval-harness

Define and run eval-driven development tests for Claude Code sessions.

Community
Advanced
ovachieverovachiever
52

lm-evaluation-harness

Evaluate language models across 60+ academic benchmarks with lm-evaluation-harness.

Community
Advanced
Chulf58Chulf58

forge:spawn

Spawn a worker session with run creation, worktree setup, and Claude Code pane.

Community
Intermediate
its-Basudebaits-Basudeba
1

eval-harness

Define and run pass/fail evaluations with regression testing for AI agents.

Community
Advanced