eval-harness

Name: eval-harness
Availability: InStock
Author: Arbisoft

Define and execute eval-driven development workflows for Claude Code tasks.

4|7|Updated Apr 9, 2026

Search Tags:#testing #metrics #evaluation #regression #claude-code #eval-driven-development #graders

Ready to run

npx skills add https://github.com/arbisoft/ai-skillforge --skill eval-harness-arbisoft

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/arbisoft/ai-skillforge/tree/main/Claude/skills/eval-harness
Command: npx skills add https://github.com/arbisoft/ai-skillforge --skill eval-harness-arbisoft

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Provides a formal framework to define, run, and track evaluations for Claude Code tasks using eval-driven development (EDD) principles, enabling objective measurement of capability and regression performance.

Core Features & Use Cases

Capability Evals to validate new features and behavior
Regression Evals to guard against unintended changes across model versions
Grader types: code-based, model-based, and human graders to balance determinism and judgment
Metrics like pass@k and pass^k to quantify reliability and stability
Eval workflow: define, implement, evaluate, and report to ship improvements with confidence

Quick Start

Define an eval, implement the checks, and run /eval check feature-name to execute and review results.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up regression tests for Claude Code tasks?▼

Regression tests for Claude Code tasks are set up using eval-driven development workflows. You define an eval, implement the checks, and run the evaluation to guard against unintended changes across model versions.

What is eval-driven development and how does it measure LLM performance?▼

Eval-driven development is a formal framework to define, run, and track evaluations for LLM tasks. It measures performance using metrics like pass@k and pass^k to quantify reliability and stability across model versions.

Can I use both deterministic code graders and model graders for task evaluation?▼

Yes, task evaluation supports deterministic code graders, model-based graders, and human graders. This combination allows you to balance objective determinism with judgment-based reviews within a structured milestone workflow.

How do I run capability tests to validate new Claude Code features?▼

Capability tests validate new features by defining an eval, implementing the checks, and executing the workflow. You run the command to execute and review results to ship improvements with confidence.

What metrics quantify reliability and stability in LLM evaluation?▼

Metrics like pass@k and pass^k quantify reliability and stability in LLM evaluation. These metrics are applied across capability and regression tests to measure performance and track improvements objectively.