eval-harness

Name: eval-harness
Availability: InStock
Author: mbadoz

Automate formal evaluation of Claude Code sessions with pass@k metrics.

2|Updated Apr 5, 2026

Search Tags:#testing #ai #evaluation #regression #benchmark #claude-code #eval

Ready to run

npx skills add https://github.com/mbadoz/mbadoz-skills --skill eval-harness-mbadoz

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/mbadoz/mbadoz-skills/tree/main/plugins/built-planner/skills/built-planner/references/skills/eval-harness
Command: npx skills add https://github.com/mbadoz/mbadoz-skills --skill eval-harness-mbadoz

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Formal evaluation framework for Claude Code sessions enabling eval-driven development by defining success criteria, running tests, and tracking regressions across model versions.

Core Features & Use Cases

Capability and regression evals with deterministic and model-based grading options.
A structured workflow (define, implement, evaluate, report) to manage eval lifecycles.
Built-in metrics (pass@k and pass^k) and baselines to quantify reliability and improvements.

Quick Start

Define an evaluation for a feature, implement the tests that exercise it, and run the eval to generate a report.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run regression tests on Claude Code sessions?▼

You can run regression tests on Claude Code sessions by defining success criteria, implementing tests, and evaluating them against baselines to track regressions across model versions.

What is eval-driven development for AI coding agents?▼

Eval-driven development for AI coding agents is a workflow that defines success criteria, runs capability and regression evals, and generates reports to drive reliable code generation.

How do I measure pass@k and pass^k metrics for AI testing benchmarks?▼

You measure pass@k and pass^k metrics for AI testing benchmarks by running formal evaluations on Claude Code sessions, using deterministic and model-based graders to quantify reliability.

Can I use model-based grading for AI evaluation instead of deterministic tests?▼

Yes, you can use model-based grading for AI evaluation instead of deterministic tests, as the framework supports multiple grader types including both deterministic and model-based options.

What's the best way to track regressions across different Claude Code model versions?▼

The best way to track regressions across Claude Code model versions is using a structured evaluation workflow that stores definitions and results in a standardized layout with regression baselines.

How do I structure an evaluation workflow for AI testing?▼

You structure an evaluation workflow for AI testing by following the define, implement, evaluate, and report lifecycle, storing eval definitions and results in a standardized layout.