skills.rest
CDO-07-New
CDO-07-NewOFFICIAL
Explore all skills by this author →
GitHub

eval-harness

Define repeatable evaluation harnesses for Claude Code task outcomes.

2|Updated Jul 1, 2026
Search Tags:#claude code#benchmarking#grading#regression-testing#evals#pass@k
One-click install
npx skills add https://github.com/CDO-07-New/TF4-AIO-03-foresight-lens-final --skill eval-harness-cdo-07-new
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/CDO-07-New/TF4-AIO-03-foresight-lens-final/tree/main/.agents/skills/eval-harness
Command: npx skills add https://github.com/CDO-07-New/TF4-AIO-03-foresight-lens-final --skill eval-harness-cdo-07-new

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you define repeatable evaluations for Claude Code workflows so you can tell whether an AI-assisted change actually works instead of relying on guesswork.

Core Features & Use Cases

  • Capability evals for validating new behaviors before implementation is considered done.
  • Regression evals for protecting existing workflows, prompts, and agent behavior from breakage.
  • Multiple grader styles including code-based checks, model-based review, and human sign-off.
  • Reliability metrics such as pass@k and pass^k for tracking consistency over time.
  • Use cases include feature validation, prompt benchmarking, release gating, and agent reliability measurement.

Quick Start

Create an eval definition for your feature, run the appropriate checks, and record the pass@k result in a report.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up regression testing for Claude Code prompt changes?▼

Benchmarking Claude Code tasks requires defining repeatable evaluation harnesses that apply code-based, model-based, or human graders to measure task outcomes. The harness tracks pass@k metrics and pass^k reliability checks across versioned baselines to quantify consistency.

What is the best way to measure AI agent reliability and consistency over time?▼

Grading Claude Code task outcomes is supported through three grader styles: code-based checks for automated validation, model-based review for AI-assisted evaluation, and human sign-off for manual review gates. You choose the appropriate grader per eval definition.

How do I create an evaluation harness for feature validation before release?▼

Creating an evaluation harness for feature validation involves defining an eval definition for the new behavior, selecting a grader type, and running checks to generate a concise evaluation report. The report records pass@k results to gate the release decision.

Can I use model-based graders for prompt benchmarking and manual review gates?▼

Yes, prompt benchmarking and manual review gates support model-based graders alongside code-based checks and human sign-off. The evaluation harness applies these grader styles to measure task outcomes and generate concise reports for release gating.

Related Skills

Explore More Software Engineering Skills→
jinyuanlujinyuanlu
3

eval-harness

Define and run eval-driven development tests for Claude Code sessions.

Community
Advanced
rongarederongarede
4

eval-harness

Evaluate Claude Code sessions with capability and regression evals.

Community
Advanced
cyphercr0wcyphercr0w
8

eval-harness

Formalize AI output evaluation in Claude Code with EDD grading mechanisms.

Community
Advanced
JaCha00JaCha00

autoresearch-goal

Run structured research workflows with Codex goal mode and OMX validation.

Community
Advanced