eval-harness

Create repeatable evaluation frameworks for Claude Code sessions with pass/fail criteria.

1|Updated Mar 18, 2026

Search Tags:#ai #evaluation #reliability #code #grading #eval

Ready to run

npx skills add https://github.com/ValenBorras/upscribe --skill eval-harness-valenborras

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/ValenBorras/upscribe/tree/main/.github/skills/eval-harness
Command: npx skills add https://github.com/ValenBorras/upscribe --skill eval-harness-valenborras

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles.

Core Features & Use Cases

Establish capability Evals to test new features and regressions to ensure stability.
Define multiple Grader Types: Code-Based, Model-Based, Human Grader to cover deterministic and open-ended checks.
Use metrics like pass@k and pass^k to quantify reliability across runs and changes.
Provide an auditable workflow and storage pattern for eval definitions and results.

Quick Start

Define an evaluation using /eval define feature-name and run checks with /eval check feature-name to begin.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up an evaluation framework for AI coding sessions?▼

Measure AI reliability by using pass@k and pass^k metrics to quantify task success across multiple runs. These metrics track regression and ensure stability when evaluating changes or comparing different model versions.

How do I grade open-ended AI code generation tasks?▼

Grade open-ended AI code tasks using model-based or human graders alongside deterministic code-based graders. This multi-grader approach covers both rigid output matching and subjective code quality assessment.

What is the best way to track AI regressions across model versions?▼

Track AI regressions across model versions by implementing an auditable storage pattern for evaluation definitions and results. This workflow maintains historical eval data to ensure stability as features evolve.

Does eval-driven development work without external dependencies?▼

Start evaluating AI reliability by defining an evaluation with /eval define feature-name, then run checks with /eval check feature-name. This establishes a repeatable framework to measure task success and track regressions.