skills.rest
andrew-starosciak
andrew-starosciakCOMMUNITY
Explore all skills by this author →
𝕏@andrewscodingGitHub

eval-harness

Evaluate Claude Code sessions with pass@k scoring and structured reports.

2|1|Updated Oct 4, 2025
Search Tags:#evaluation#Claude Code#quality-assurance#EDD#testing-framework#code-evaluation
One-click install
npx skills add https://github.com/andrew-starosciak/deep-algo --skill eval-harness-andrew-starosciak
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/andrew-starosciak/deep-algo/tree/main/.claude/docs/zh-TW/skills/eval-harness
Command: npx skills add https://github.com/andrew-starosciak/deep-algo --skill eval-harness-andrew-starosciak

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill provides a formal evaluation framework for Claude Code sessions, enabling eval-driven development (EDD) by defining objective evaluation criteria, executing continuous checks, and tracking regressions.

Core Features & Use Cases

  • Capability Evals: tests Claude capabilities to verify new functionality.
  • Regression Evals: ensures changes do not break existing features by comparing against baselines.
  • Quantitative Metrics: uses pass@k and pass^k scoring to quantify reliability, plus model and human grader prompts for comprehensive assessment.
  • Reporting & Artifacts: produces structured eval reports and stores artifacts under the .claude/evals directory for traceability.

Quick Start

a. Define an eval: create a file under .claude/evals/feature-name.md with the evaluation criteria. b. Implement checks: add the necessary evaluation steps and prompts. c. Run evaluation: /eval check feature-name d. Generate report: /eval report feature-name

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is eval-driven development for Claude Code?▼

Eval-driven development (EDD) for Claude Code uses a formal evaluation framework to define objective criteria, execute continuous checks, and track regressions across code changes, models, and human reviews.

How do I set up regression evals for Claude Code sessions?▼

To set up regression evals, create a Markdown file under the .claude/evals/ directory, implement the necessary evaluation steps and prompts, run /eval check, and generate a report with /eval report.

How does pass@k and pass^k scoring work for code evaluation?▼

Pass@k and pass^k scoring are quantitative metrics used to quantify the reliability of Claude Code capabilities, providing objective assessment data through model and human grader prompts.

Can I track baseline regressions when testing new Claude Code functionality?▼

Yes, capability and regression evals compare new functionality against established baselines to verify features and ensure code changes do not break existing capabilities.

Do I need external dependencies to run the Claude Code evaluation framework?▼

No external dependencies are required to run the evaluation framework. The tooling operates entirely within Claude Code sessions and stores all evaluation artifacts locally under the .claude/evals directory.

Related Skills

Explore More Software Engineering Skills→
ysyecustysyecust
40

eval-harness

Codify and automate Claude Code session evaluations with pass@k metrics.

Community
Intermediate
gugug168gugug168

eval-harness

Define and run capability and regression evaluations for Claude Code sessions.

Community
Advanced
TruCol270TruCol270
1

eval-harness

Evaluate Claude Code sessions with configurable graders and pass@k metrics.

Community
Advanced
ThejanaJayalathThejanaJayalath

configure-ecc

Automate deployment and verification of Everything Claude Code skills and rule sets.

Community
Advanced