skills.rest
derekhu0002
derekhu0002COMMUNITY
Explore all skills by this author →
WebsiteGitHub

eval-harness

Automates eval-driven development for Claude Code sessions with pass@k metrics and regression suites.

Updated Mar 1, 2026
Search Tags:#claude-code#grading#pass-at-k#ai-evaluation#regression-tests#eval-driven-development
One-click install
npx skills add https://github.com/derekhu0002/ai4pb-orchestrator --skill eval-harness-derekhu0002
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/derekhu0002/ai4pb-orchestrator/tree/main/.agents/skills/eval-harness
Command: npx skills add https://github.com/derekhu0002/ai4pb-orchestrator --skill eval-harness-derekhu0002

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Formalizes evaluation for Claude Code sessions by turning design goals into repeatable tests and metrics that guide development.

Core Features & Use Cases

  • Capability Evals: define and run tests that verify new features or capabilities.
  • Regression Evals: ensure changes don't break existing behavior across model iterations.
  • Pass@k Metrics & Grading: measure reliability with structured scoring and recorded results.
  • Evaluation Workflow & Storage: track eval definitions, runs, and reports in a consistent format.

Quick Start

Define eval criteria, implement tests, run the evaluation workflow, and review the results.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up regression tests for Claude Code sessions?▼

You can automate eval-driven development for Claude Code by defining capability and regression evals, implementing tests, running the evaluation workflow, and reviewing the stored results.

What is pass@k metric evaluation in AI-driven development?▼

Pass@k metric evaluation measures AI reliability by applying structured scoring to recorded results, verifying that a capability or regression test passes within a specified number of attempts.

Can I use both code-based and model-based graders for AI evaluation?▼

Yes, the evaluation harness supports code-based graders, model-based graders, and human reviews to assess test results and measure reliability across different model versions.

How does eval-driven development store evaluation definitions and reports?▼

Eval-driven development stores evaluation definitions, runs, and reports using a standard JSON and Markdown format, ensuring consistent tracking and review of the evaluation workflow.

Does Claude Code eval-driven development work without external dependencies?▼

Yes, the eval-driven harness operates without external dependencies, allowing you to formalize evaluation for Claude Code sessions directly and turn design goals into repeatable metrics.

Related Skills

Explore More Software Engineering Skills→
jinyuanlujinyuanlu
3

eval-harness

Define and run eval-driven development tests for Claude Code sessions.

Community
Advanced
fredericvahrenhorstfredericvahrenhorst

eval-harness

Run capability and regression evals for Claude Code sessions with pass@k metrics.

Community
Advanced
rivaldiekaptrrrrivaldiekaptrrr

eval-harness

Define and run eval suites to validate AI task performance.

Community
Advanced
GatsbyhateyouGatsbyhateyou

eval-harness

Define, execute, and report capability and regression evaluations using pass@k and pass^k metrics.

Community
Advanced