eval-harness

Define, execute, and report capability and regression evals for Claude Code sessions.

1.8k|303|Updated Jan 27, 2026

Search Tags:#workflow #Claude Code #eval #software-testing #evaluation-framework #eval-driven

Ready to run

npx skills add https://github.com/xu-xiang/everything-claude-code-zh --skill eval-harness-xu-xiang

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/xu-xiang/everything-claude-code-zh/tree/main/docs/zh-TW/skills/eval-harness
Command: npx skills add https://github.com/xu-xiang/everything-claude-code-zh --skill eval-harness-xu-xiang

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill provides a formal evaluation framework for Claude Code sessions, enabling eval-driven development by defining clear evaluation criteria before implementation and continuously validating changes through structured evals.

Core Features & Use Cases

Capability and regression evals to validate new features and prevent regressions.
Multiple grader options (code-based, model-based, and human) to score changes.
Integrated workflow from eval definition to reporting, with metrics like pass@k and pass^k for reliability.

Quick Start

Create an eval definition under .claude/evals (e.g., add-authentication.md), then run: /eval define add-authentication /eval check add-authentication /eval report add-authentication

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up eval-driven development for Claude Code sessions?▼

Eval-driven development for Claude Code requires creating standardized eval definitions in .claude/evals, then running commands to execute checks and generate reports with pass@k and pass^k metrics.

What are capability and regression evals in an evaluation framework?▼

Capability and regression evals validate new features and prevent regressions by continuously checking changes against structured evaluation criteria defined before implementation.

How do I run model-based and code-based graders for software testing evals?▼

Software testing evals support multiple grader types including code-based, model-based, and human graders to score changes within the integrated workflow from definition to reporting.

Can I define custom evaluation criteria before implementation in Claude Code?▼

Yes, you can define custom evaluation criteria before implementation by creating a standardized eval definition file under .claude/evals and executing it through the formal workflow commands.

What is the best way to measure pass@k reliability metrics for Claude Code evaluations?▼

Measuring pass@k and pass^k reliability metrics is handled natively by the eval-harness framework, which generates these metrics automatically when you run the report command on your eval definitions.