Verified:GitHub

eval-harness

Codify and automate Claude Code session evaluations with pass@k metrics.

40|8|Updated Jan 23, 2026

Search Tags:#Claude Code #reporting #eval-driven development #capability eval #regression eval #pass@k #evaluation framework

Ready to run

npx skills add https://github.com/ysyecust/everything-claude-code --skill eval-harness-ysyecust

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/ysyecust/everything-claude-code/tree/main/docs/zh-TW/skills/eval-harness
Command: npx skills add https://github.com/ysyecust/everything-claude-code --skill eval-harness-ysyecust

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill provides a formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) to define, execute, and monitor evaluations throughout development.

Core Features & Use Cases

Eval types: Capability evals and Regression evals to validate new features and preserve existing behavior.
Scoring & thresholds: Pass@k and pass^k metrics, different grader types (code-based, model-based, human) for robust assessment.
Workflow & tooling: Predefine evals in .claude/evals, run /eval check, and publish reports with /eval report.

Quick Start

Define: Create an eval file at .claude/evals/feature-name.md.
Implement: Write eval definitions and tests.
Run: Use /eval check feature-name and /eval report feature-name to generate a summary.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run regression and capability evaluations for Claude Code sessions?▼

You can run regression and capability evaluations for Claude Code sessions by defining eval files in .claude/evals and executing /eval check to validate behavior and generate standardized reports.

What is eval-driven development and how does it apply to Claude Code?▼

Eval-driven development for Claude Code involves defining evaluations before coding, executing capability and regression checks, and monitoring pass@k metrics to ensure feature validity and prevent behavioral regressions.

How do I generate standardized evaluation reports for Claude Code features?▼

To generate standardized evaluation reports for Claude Code features, implement your eval tests in a markdown file under .claude/evals, then execute the /eval report command to produce a summary of pass@k metrics.

Does the eval framework support pass@k metrics and model-based graders?▼

Yes, the eval framework supports pass@k and pass^k scoring metrics, alongside code-based, model-based, and human grader types to provide robust assessment for Claude Code capability and regression evaluations.

What are the limitations of using markdown files for Claude Code regression evals?▼

Using markdown files under .claude/evals for regression evals requires predefined eval definitions and tests before execution, meaning evaluations must be codified manually rather than generated dynamically during the Claude Code session.