skills.rest
Maelwalser
MaelwalserCOMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Implement formal capability and regression evals for Claude Code session outputs.

Updated Aug 27, 2026
Search Tags:#claude-code#regression-testing#eval#grader#pass@k#eval-driven-development#agent-evaluation
One-click install
npx skills add https://github.com/Maelwalser/claude-config --skill eval-harness-maelwalser
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/Maelwalser/claude-config/tree/main/skills/eval-harness
Command: npx skills add https://github.com/Maelwalser/claude-config --skill eval-harness-maelwalser

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Provides a formal, repeatable evaluation framework that replaces ad-hoc judgement with measurable pass/fail criteria, preventing regressions and quantifying agent reliability across development cycles.

Core Features & Use Cases

  • Eval-driven development: Define capability and regression evals before implementation and treat them as unit tests for AI behavior.
  • Multiple grader types: Support deterministic code graders, model-based rubrics, and human review for ambiguous or high-risk checks.
  • Metrics and reporting: Track pass@k and pass^k metrics, store eval definitions and run histories, and generate concise eval reports for CI and release gating.
  • Use Case: Create a suite of capability evals for a new authentication feature, run them during development, and block releases if regression evals fail.

Quick Start

Create an eval definition for "feature-xyz" under .claude/evals/feature-xyz.md and run the eval check to generate a pass/fail report and metrics.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up regression testing for Claude Code agent outputs?▼

Regression testing for Claude Code agents is configured by creating eval definitions under .claude/evals/ and running an eval check to generate pass/fail reports and pass@k metrics.

What is eval-driven development and how does it apply to AI agents?▼

Eval-driven development for AI agents involves defining capability and regression evals before implementation, treating them as unit tests for AI behavior to prevent regressions and quantify reliability across development cycles.

Can I use deterministic code graders and model-based rubrics together for agent evaluation?▼

Yes, agent evaluation supports multiple grader types simultaneously, including deterministic code graders, model-based rubrics, and human review flags to handle both strict and ambiguous checks.

How do I calculate pass@k and pass^k metrics for AI benchmarking in CI pipelines?▼

Pass@k and pass^k metrics for AI benchmarking are calculated by running evaluation checks that track these metrics, store run histories, and generate concise eval reports specifically for CI and release gating.

Does this evaluation framework work for blocking releases if automated agent tests fail?▼

Yes, the evaluation framework works for blocking releases by running regression evals during development and generating concise reports that can gate releases if capability evals fail.

When do I need human review flags instead of deterministic graders for agent benchmarking?▼

Human review flags are needed for agent benchmarking when handling ambiguous or high-risk checks that deterministic code graders and model-based rubrics cannot reliably evaluate.

Related Skills

Explore More Software Engineering Skills→
rongarederongarede
4

eval-harness

Evaluate Claude Code sessions with capability and regression evals.

Community
Advanced
vinayakgvinayakg

eval-harness

Implement eval-driven development with code-based, model-based, and human grading.

Community
Advanced
ldk-hubldk-hub

eval-harness

Automate eval-driven development workflows for Claude Code sessions.

Community
Advanced
linnefromicelinnefromice

eval-harness

Evaluate Claude Code sessions with code-based, model-based, and human graders.

Community
Advanced