eval-harness

Define and run evaluations for Claude Code changes with structured graders and reports.

Updated Jan 30, 2026

Search Tags:#testing #metrics #evaluation #framework #claude-code #eval-driven #graders

Ready to run

npx skills add https://github.com/GoBeromsu/My-Awesome-RA --skill eval-harness-goberomsu

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/GoBeromsu/My-Awesome-RA/tree/main/.claude/skills/eval-harness
Command: npx skills add https://github.com/GoBeromsu/My-Awesome-RA --skill eval-harness-goberomsu

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Solves the problem of validating Claude Code changes by providing a formal evaluation framework.

Core Features & Use Cases

Capability and Regression eval definitions to codify expected behavior.
Deterministic and model-based graders with structured scoring.
Evaluation storage, reports, and baselines to track changes.
Workflow stages: Define, Implement, Evaluate, and Report.

Quick Start

Define at least one eval, implement the changes, and run the evaluation to generate a report.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I validate Claude Code changes without manual testing?▼

You can validate Claude Code changes using an eval-driven testing framework that defines capability and regression evaluations, runs multiple graders, and outputs repeatable reports to track baselines.

What is an eval-driven testing framework for AI code generation?▼

An eval-driven testing framework formalizes AI code validation through structured eval definitions, deterministic and model-based graders, and tracked metrics like pass@k and pass^k to generate repeatable evaluation reports.

How do I measure pass@k and pass^k metrics for code generation models?▼

You measure pass@k and pass^k metrics by defining capability evals, running them through code-based or model-based graders, and generating structured evaluation reports across Define, Implement, Evaluate, and Report stages.

Can I use both deterministic and model-based graders for regression testing?▼

Yes, you can use deterministic code-based graders alongside model-based graders to score regression and capability evals, ensuring structured scoring and repeatable evaluation baselines for your changes.

What's the best way to track evaluation baselines for AI coding changes?▼

The best way to track evaluation baselines is to use a formal evaluation framework that enforces structured storage of eval definitions and results, comparing new runs against stored baselines in a final report.

Why do I need a formal evaluation framework instead of ad hoc testing?▼

A formal evaluation framework solves the problem of unrepeatable validation by codifying expected behavior into regression evals, applying structured graders, and storing baselines to ensure informed decision-making.