skills.rest
akirschke15-cmd
akirschke15-cmdCOMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Define and run capability and regression evals for Claude Code sessions.

Updated Mar 28, 2026
Search Tags:#prompt engineering#reliability#quality gates#regression testing#evals#pass@k#agent grading
One-click install
npx skills add https://github.com/akirschke15-cmd/Cato-Registry --skill eval-harness-akirschke15-cmd
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/akirschke15-cmd/Cato-Registry/tree/main/.claude/skills/eval-harness
Command: npx skills add https://github.com/akirschke15-cmd/Cato-Registry --skill eval-harness-akirschke15-cmd

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Eval-harness fixes the problem of shipping AI-assisted changes without objective proof that the agent meets requirements, stays reliable, and doesn’t regress over time.

Core Features & Use Cases

  • Eval-driven development workflow: define expected behavior before implementation, run evals during development, and track regressions as prompts/agents change.
  • Capability and regression evaluations: create capability evals to validate new behavior and regression evals to confirm existing functionality remains intact.
  • Grader options for different certainty levels: use deterministic code-based graders, rubric-style model-based graders, and human review gates for ambiguous or security-sensitive outcomes.

Use cases: designing release gates for prompt/agent updates, building regression suites for EDD workflows, benchmarking reliability across model versions using pass@k/pass^k metrics, and producing structured eval reports for stakeholder review.

Quick Start

Ask the AI to create an EDD eval definition for feature authentication and include capability, regression, and pass@3/pass^3 success thresholds.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is eval-driven development and how does it prevent AI agent regressions?▼

Eval-driven development defines expected AI behavior before implementation and runs regression evals to confirm existing functionality remains intact. It prevents shipping prompt or agent updates that degrade reliability over time.

How do I set up regression testing for prompt engineering changes?▼

Create regression evals to validate existing functionality, define expected behavior before coding, and run graded checks using deterministic code-based graders, rubric-style model-based graders, or human review gates for security-sensitive outcomes.

How do pass@k and pass^k metrics measure AI reliability across model versions?▼

Pass@k and pass^k metrics benchmark reliability by tracking graded eval outcomes across model versions. They provide structured success thresholds to quantify whether capability and regression evaluations meet requirements during prompt or agent updates.

Can I use model-based graders for ambiguous agent outputs instead of deterministic code?▼

Yes, you can use rubric-style model-based graders for ambiguous outcomes, deterministic code-based graders for high certainty, and human review gates for security-sensitive results. This tiered grading approach ensures objective validation across different evaluation scenarios.

What YAML metadata is required for discoverability in Claude Code eval sessions?▼

YAML metadata is required to establish a formal evaluation framework for Claude Code sessions. It enables discoverability and supports operational procedures for defining evals, running graded checks, generating structured reports, and tracking pass@k metrics.

When should I use capability evals versus regression evals?▼

Use capability evals to validate new behavior and regression evals to confirm existing functionality remains intact. Both are needed to build comprehensive release gates for prompt or agent updates and prevent unvalidated AI behavior changes.

Related Skills

Explore More Software Engineering Skills→
PabloGarciaFuentesPabloGarciaFuentes

astro-framework

Build Astro islands with client directives, Zod schemas, and SSR adapters.

Community
Advanced
ValenBorrasValenBorras
1

eval-harness

Create repeatable evaluation frameworks for Claude Code sessions with pass/fail criteria.

Community
Advanced
cescraflicescrafli

eval-harness

Automate Claude Code session evaluations with code, model, and human graders.

Community
Advanced
sayasaya8039sayasaya8039
2

eval-harness

Define pass/fail criteria and run regression evals for Claude Code agents.

Community
Advanced