eval-harness

Evaluate Claude Code sessions with pass@k scoring and structured reports.

2|1|Updated Oct 4, 2025

Search Tags:#evaluation #Claude Code #quality-assurance #EDD #testing-framework #code-evaluation

Ready to run

npx skills add https://github.com/andrew-starosciak/deep-algo --skill eval-harness-andrew-starosciak

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/andrew-starosciak/deep-algo/tree/main/.claude/docs/zh-TW/skills/eval-harness
Command: npx skills add https://github.com/andrew-starosciak/deep-algo --skill eval-harness-andrew-starosciak

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill provides a formal evaluation framework for Claude Code sessions, enabling eval-driven development (EDD) by defining objective evaluation criteria, executing continuous checks, and tracking regressions.

Core Features & Use Cases

Capability Evals: tests Claude capabilities to verify new functionality.
Regression Evals: ensures changes do not break existing features by comparing against baselines.
Quantitative Metrics: uses pass@k and pass^k scoring to quantify reliability, plus model and human grader prompts for comprehensive assessment.
Reporting & Artifacts: produces structured eval reports and stores artifacts under the .claude/evals directory for traceability.

Quick Start

a. Define an eval: create a file under .claude/evals/feature-name.md with the evaluation criteria. b. Implement checks: add the necessary evaluation steps and prompts. c. Run evaluation: /eval check feature-name d. Generate report: /eval report feature-name

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is eval-driven development for Claude Code?▼

Eval-driven development (EDD) for Claude Code uses a formal evaluation framework to define objective criteria, execute continuous checks, and track regressions across code changes, models, and human reviews.

How do I set up regression evals for Claude Code sessions?▼

To set up regression evals, create a Markdown file under the .claude/evals/ directory, implement the necessary evaluation steps and prompts, run /eval check, and generate a report with /eval report.

How does pass@k and pass^k scoring work for code evaluation?▼

Pass@k and pass^k scoring are quantitative metrics used to quantify the reliability of Claude Code capabilities, providing objective assessment data through model and human grader prompts.

Can I track baseline regressions when testing new Claude Code functionality?▼

Yes, capability and regression evals compare new functionality against established baselines to verify features and ensure code changes do not break existing capabilities.

Do I need external dependencies to run the Claude Code evaluation framework?▼

No external dependencies are required to run the evaluation framework. The tooling operates entirely within Claude Code sessions and stores all evaluation artifacts locally under the .claude/evals directory.