skills.rest
songkg7
songkg7COMMUNITY
Explore all skills by this author →
WebsiteGitHub

eval-harness

Automate Claude Code session evaluations with code-based, model-based, and human graders.

6|Updated Mar 25, 2023
Search Tags:#metrics#evaluation#Claude Code#grading#eval-driven development#capability eval#regression eval
One-click install
npx skills add https://github.com/songkg7/dotfiles --skill eval-harness-songkg7
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/songkg7/dotfiles/tree/main/dot_claude/skills/eval-harness
Command: npx skills add https://github.com/songkg7/dotfiles --skill eval-harness-songkg7

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill provides a structured framework to define and run evaluations for Claude Code changes, ensuring predictable, test-driven progress with eval-driven development practices.

Core Features & Use Cases

  • Capability Evals: define and run feature checks before coding.
  • Regression Evals: verify existing behavior remains intact after changes.
  • Graders: code-based, model-based, and human graders for flexibility.
  • Metrics & Reporting: pass@k, pass^k, and evaluation reports.

Use cases:

  • Ensure reliability in AI code sessions
  • Validate feature adoption before shipping
  • Document evaluation results for audits and collaboration.

Quick Start

Install and configure the eval harness in your project. Define evals in .claude/evals and run checks with the provided commands. Review EVAL REPORTS to ship-ready results.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run regression evaluations on Claude Code sessions?▼

The harness supports code-based, model-based, and human graders for evaluating Claude Code sessions, allowing you to choose the appropriate grading method for each specific capability or regression check.

What metrics are collected for eval-driven development in AI coding?▼

The eval-driven development framework collects pass@k and pass^k metrics during Claude Code session evaluations, generating structured reports that document whether features pass capability and regression checks before shipping.

How do I set up capability evals before coding new features?▼

Review the generated evaluation reports to confirm ship-ready results. These structured reports document pass@k and pass^k metrics, validating feature adoption and providing audit trails for collaboration.

Can I use model-based graders alongside code-based graders for AI session evaluation?▼

Yes, the evaluation framework supports code-based, model-based, and human graders simultaneously, providing flexibility to apply different grading strategies across various project workflows and evaluation scenarios.

What is eval-driven development and when do I need it for Claude Code changes?▼

Eval-driven development is a structured practice of defining and running capability and regression evaluations before and after Claude Code changes, needed when you require predictable, test-driven progress in AI coding sessions.

Related Skills

Explore More Software Engineering Skills→
rongarederongarede
4

eval-harness

Evaluate Claude Code sessions with capability and regression evals.

Community
Advanced
oabdelmaksoudoabdelmaksoud
3

eval-harness

Implement evaluation-driven development with code and model-based evaluators.

Community
Advanced
RyanJarvRyanJarv

Hooks Automation

Automate Claude Code operations with pre/post-operation hooks and session management.

Community
Advanced
j7-devj7-dev
19

eval-harness

Formalizes evaluation frameworks for AI-assisted development using eval-driven development principles.

Community
Advanced