skills.rest
codelably
codelablyOFFICIAL
Explore all skills by this author →
GitHub

eval-harness

Automate formal evaluation workflows for Claude Code sessions.

41|8|Updated Feb 7, 2026
Search Tags:#testing#evaluation#reporting#claude-code#eval-driven#regression-evals
One-click install
npx skills add https://github.com/codelably/harmony-claude-code --skill eval-harness-codelably
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/codelably/harmony-claude-code/tree/main/docs/zh-TW/skills/eval-harness
Command: npx skills add https://github.com/codelably/harmony-claude-code --skill eval-harness-codelably

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This evaluation framework provides a formal, repeatable way to define, execute, and review evals for Claude Code sessions, enabling eval-driven development.

Core Features & Use Cases

  • Define capability and regression evals that codify expected behavior and regression boundaries.
  • Run evals during development and production to monitor quality with metrics like pass@k and pass^k.
  • Store all eval definitions and results in a centralized, auditable location under .claude/evals for traceability.
  • Support multiple grader types (code-based, model-based, and human review) to balance automation with safety.

Quick Start

Create an eval named feature-xyz under .claude/evals, implement the eval definitions and a baseline, run /eval check feature-xyz, and review the resulting report.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up eval-driven development for Claude Code sessions?▼

Eval-driven development for Claude Code is set up by creating eval definitions under the .claude/evals directory. You implement a baseline, run the /eval check command, and review the generated report to monitor quality.

What are pass@k and pass^k metrics in regression evals?▼

Pass@k and pass^k are evaluation metrics used to monitor quality during development and production. They measure capability and regression boundaries by tracking pass rates across multiple eval runs for Claude Code tasks.

Can I use model-based graders for automated evaluation reporting?▼

Yes, automated evaluation reporting supports code-based, model-based, and human review graders. This balances automation with safety by allowing model graders to evaluate Claude Code session outputs automatically.

How do I run regression evals to monitor AI agent quality?▼

Regression evals are executed via the /eval check command after defining them under .claude/evals. The framework persists results with pass@k and pass^k metrics, generating auditable reports for quality monitoring.

Does eval-driven testing work without external dependencies?▼

Yes, the eval harness operates with no external dependencies. It defines and persists eval definitions and results locally under .claude/evals, providing a centralized and auditable location for traceability.

When do I need formal capability tests for Claude Code tasks?▼

Formal capability tests are needed when AI development teams build agents and tools requiring eval-driven quality controls. They codify expected behavior and regression boundaries for Claude Code sessions.

Related Skills

Explore More Software Engineering Skills→
furrycolombia-sysfurrycolombia-sys
1

storybook

Initialize and configure Storybook tasks using the Claude workflow.

Community
Intermediate
oabdelmaksoudoabdelmaksoud
3

eval-harness

Implement evaluation-driven development with code and model-based evaluators.

Community
Advanced
sunnypatneedisunnypatneedi
11

ai-evaluation

Define and run structured evaluation suites for AI/LLM systems.

Community
Advanced
rongarederongarede
4

eval-harness

Evaluate Claude Code sessions with capability and regression evals.

Community
Advanced