eval-harness

Coordinate eval-driven development workflows for Claude Code sessions.

Updated Mar 21, 2026

Search Tags:#testing #ai #evaluation #regression #eval #pass@k

Ready to run

npx skills add https://github.com/hieuck/Pro5ChromeManager --skill eval-harness-hieuck

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/hieuck/Pro5ChromeManager/tree/main/.agents/skills/eval-harness
Command: npx skills add https://github.com/hieuck/Pro5ChromeManager --skill eval-harness-hieuck

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Evaluates Claude Code sessions using a formal evaluation framework to ensure reliable model behavior and traceable results.

Core Features & Use Cases

Capability and regression eval templates to measure model performance.
Grader types (code-based, model-based, human) to tailor evaluation pipelines.
Metrics and workflows to define, implement, and report evaluation results.

Quick Start

Define a new evaluation with /eval define feature and then run checks with /eval check feature to start the evaluation pipeline.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up an evaluation pipeline for Claude Code sessions?▼

You can set up an evaluation pipeline by defining a new evaluation with the /eval define command and running checks with /eval check to measure session reliability.

What is pass@k metric and how does it measure AI model reliability?▼

Pass@k is a standardized metric used to measure the probability that at least one successful response is generated within k attempts, quantifying AI model reliability for regression testing.

Can I use human graders alongside code-based and model-based grading?▼

Yes, the evaluation framework supports multiple grader types including code-based, model-based, and human graders to tailor the evaluation pipeline to your specific requirements.

How do I build a regression suite across different AI model versions?▼

You can build regression suites by applying capability and regression evaluation templates to measure and compare model performance across different versions using standardized metrics.

What's the best way to define pass/fail criteria for an AI testing workflow?▼

The best way to define pass/fail criteria is by establishing a formal evaluation environment that applies standardized metrics and workflows to ensure reliable, traceable model behavior.

Why do I need formal capability and regression eval definitions for AI testing?▼

You need formal eval definitions to ensure reliable model behavior and traceable results, preventing regressions and systematically measuring AI capabilities across development iterations.