eval-harness

Name: eval-harness
Availability: InStock
Author: Aayushsoam

Evaluate Claude Code sessions with pass/fail criteria and regression metrics.

1|Updated Mar 31, 2026

Search Tags:#testing #framework #claude-code #model-evaluation #eval-driven #regression-tests

Ready to run

npx skills add https://github.com/aayushsoam/clawbot-plus --skill eval-harness-aayushsoam

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/aayushsoam/clawbot-plus/tree/main/skills/eval-harness
Command: npx skills add https://github.com/aayushsoam/clawbot-plus --skill eval-harness-aayushsoam

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Formalizes evaluation for Claude Code sessions by providing an eval-driven framework that defines success criteria, runs tests, and records results to guide development workflows.

Core Features & Use Cases

End-to-end eval workflow (define, implement, evaluate, report) for AI-driven coding tasks.
Supports capability and regression evals with pass@k and pass^k style metrics.
Provides deterministic code-based graders and model-based rubrics to ensure reliability across model versions.

Quick Start

Run the evaluation workflow to define your first eval and generate a report.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up regression tests for Claude Code model versions?▼

Regression tests for Claude Code are formalized using an eval-driven framework that defines success criteria, runs tests, and records results to track performance across model versions.

What is eval-driven testing for AI coding tasks?▼

Eval-driven testing for AI coding tasks defines success criteria, runs tests, and records results to guide development workflows using pass@k and pass^k metrics.

How do I evaluate Claude Code agent interactions deterministically?▼

Claude Code agent interactions are evaluated deterministically by applying code-based graders and model-based rubrics to ensure reliable, reproducible pass/fail assessments.

Can I use pass@k metrics for Claude Code prompt evaluation?▼

Yes, pass@k and pass^k style metrics are supported for Claude Code prompt evaluation, measuring capability and regression performance across model versions.

Does the eval-driven framework support end-to-end reporting for coding sessions?▼

The eval-driven framework supports end-to-end reporting for coding sessions by guiding a full workflow from define to report, implementing tests, evaluating results, and generating reports.

What's the best way to standardize Claude Code session assessments?▼

Standardizing Claude Code session assessments is best achieved using an eval-driven framework that defines eval types, graders, and metrics to produce deterministic, reproducible results.