eval-harness

Automates eval-driven development for Claude Code sessions with pass@k metrics and regression suites.

Updated Mar 1, 2026

Search Tags:#claude-code #grading #pass-at-k #ai-evaluation #regression-tests #eval-driven-development

Ready to run

npx skills add https://github.com/derekhu0002/ai4pb-orchestrator --skill eval-harness-derekhu0002

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/derekhu0002/ai4pb-orchestrator/tree/main/.agents/skills/eval-harness
Command: npx skills add https://github.com/derekhu0002/ai4pb-orchestrator --skill eval-harness-derekhu0002

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Formalizes evaluation for Claude Code sessions by turning design goals into repeatable tests and metrics that guide development.

Core Features & Use Cases

Capability Evals: define and run tests that verify new features or capabilities.
Regression Evals: ensure changes don't break existing behavior across model iterations.
Pass@k Metrics & Grading: measure reliability with structured scoring and recorded results.
Evaluation Workflow & Storage: track eval definitions, runs, and reports in a consistent format.

Quick Start

Define eval criteria, implement tests, run the evaluation workflow, and review the results.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up regression tests for Claude Code sessions?▼

You can automate eval-driven development for Claude Code by defining capability and regression evals, implementing tests, running the evaluation workflow, and reviewing the stored results.

What is pass@k metric evaluation in AI-driven development?▼

Pass@k metric evaluation measures AI reliability by applying structured scoring to recorded results, verifying that a capability or regression test passes within a specified number of attempts.

Can I use both code-based and model-based graders for AI evaluation?▼

Yes, the evaluation harness supports code-based graders, model-based graders, and human reviews to assess test results and measure reliability across different model versions.

How does eval-driven development store evaluation definitions and reports?▼

Eval-driven development stores evaluation definitions, runs, and reports using a standard JSON and Markdown format, ensuring consistent tracking and review of the evaluation workflow.

Does Claude Code eval-driven development work without external dependencies?▼

Yes, the eval-driven harness operates without external dependencies, allowing you to formalize evaluation for Claude Code sessions directly and turn design goals into repeatable metrics.