eval-harness

Define and run capability and regression evals for Claude Code sessions.

Updated Mar 24, 2026

Search Tags:#workflow #evaluation #prompt-engineering #ai-testing #model-evaluation #devtooling #evals

Ready to run

npx skills add https://github.com/Oruga420/claude-code-skills --skill eval-harness-oruga420

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/Oruga420/claude-code-skills/tree/main/eval-harness
Command: npx skills add https://github.com/Oruga420/claude-code-skills --skill eval-harness-oruga420

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.

Core Features & Use Cases

Capability Evals and Regression Evals to validate AI behavior
Graders: code-based, model-based, and human evaluators for flexible scoring
Metrics: pass@k and pass^k with a centralized eval registry
End-to-end workflow: define, implement, evaluate, and report
Storage pattern: store definitions and logs under .claude/evals

Quick Start

Define your first eval with /eval define feature-name and then implement, run checks, and generate a report with /eval report feature-name.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up AI model evaluation for Claude Code sessions?▼

To set up AI model evaluation for Claude Code sessions, define your first eval with /eval define feature-name, then implement, run checks, and generate a report using /eval report feature-name.

What is eval-driven development and how does it apply to prompt engineering?▼

Eval-driven development for prompt engineering validates AI behavior through capability and regression evals, enforcing pass@k and pass^k metrics to improve development reliability across model updates.

Can I use human graders alongside automated code graders for AI testing?▼

Yes, the AI testing framework supports code-based, model-based, and human evaluators as graders, providing flexible scoring options for validating Claude Code session behavior.

What's the best way to prevent regressions when updating AI workflows?▼

The best way to prevent regressions in AI workflows is implementing regression evals that enforce pass@k and pass^k metrics, storing definitions and logs under the .claude/evals directory.

Does the evaluation framework require external dependencies to run capability evals?▼

No external dependencies are required to run capability evals; the framework operates self-contained within Claude Code, managing eval definitions and logs under the .claude/evals storage pattern.

Why define evals before implementing features in an AI testing workflow?▼

Defining evals before implementing features establishes clear capability and regression benchmarks, ensuring the end-to-end workflow from definition to reporting maintains AI development reliability.