eval-harness

Name: eval-harness
Availability: InStock
Author: codelably

Automate formal evaluation workflows for Claude Code sessions.

41|8|Updated Feb 7, 2026

Search Tags:#testing #evaluation #reporting #claude-code #eval-driven #regression-evals

Ready to run

npx skills add https://github.com/codelably/harmony-claude-code --skill eval-harness-codelably

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/codelably/harmony-claude-code/tree/main/docs/zh-TW/skills/eval-harness
Command: npx skills add https://github.com/codelably/harmony-claude-code --skill eval-harness-codelably

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This evaluation framework provides a formal, repeatable way to define, execute, and review evals for Claude Code sessions, enabling eval-driven development.

Core Features & Use Cases

Define capability and regression evals that codify expected behavior and regression boundaries.
Run evals during development and production to monitor quality with metrics like pass@k and pass^k.
Store all eval definitions and results in a centralized, auditable location under .claude/evals for traceability.
Support multiple grader types (code-based, model-based, and human review) to balance automation with safety.

Quick Start

Create an eval named feature-xyz under .claude/evals, implement the eval definitions and a baseline, run /eval check feature-xyz, and review the resulting report.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up eval-driven development for Claude Code sessions?▼

Eval-driven development for Claude Code is set up by creating eval definitions under the .claude/evals directory. You implement a baseline, run the /eval check command, and review the generated report to monitor quality.

What are pass@k and pass^k metrics in regression evals?▼

Pass@k and pass^k are evaluation metrics used to monitor quality during development and production. They measure capability and regression boundaries by tracking pass rates across multiple eval runs for Claude Code tasks.

Can I use model-based graders for automated evaluation reporting?▼

Yes, automated evaluation reporting supports code-based, model-based, and human review graders. This balances automation with safety by allowing model graders to evaluate Claude Code session outputs automatically.

How do I run regression evals to monitor AI agent quality?▼

Regression evals are executed via the /eval check command after defining them under .claude/evals. The framework persists results with pass@k and pass^k metrics, generating auditable reports for quality monitoring.

Does eval-driven testing work without external dependencies?▼

Yes, the eval harness operates with no external dependencies. It defines and persists eval definitions and results locally under .claude/evals, providing a centralized and auditable location for traceability.

When do I need formal capability tests for Claude Code tasks?▼

Formal capability tests are needed when AI development teams build agents and tools requiring eval-driven quality controls. They codify expected behavior and regression boundaries for Claude Code sessions.