skills.rest
mthang1801
mthang1801COMMUNITY
Explore all skills by this author →
WebsiteGitHub

eval-harness

Define, run, and assess AI code sessions with structured evals.

Updated Aug 27, 2026
Search Tags:#automation#testing#ai#evaluation#code-review#grading#eval
One-click install
npx skills add https://github.com/mthang1801/go-domain-driven-design --skill eval-harness-mthang1801
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/mthang1801/go-domain-driven-design/tree/main/.claude/skills/eval-harness
Command: npx skills add https://github.com/mthang1801/go-domain-driven-design --skill eval-harness-mthang1801

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Automate and standardize the evaluation of AI code sessions by providing a deterministic framework for defining tests, running them, and recording results.

Core Features & Use Cases

  • Capability Evals: predefine functional goals and verify AI outputs against criteria.
  • Regression Evals: safeguard against regressions across changes with structured checks.
  • Metrics & Reports: generate pass@k and pass^k style reports to quantify reliability.

Quick Start

Install and configure the eval harness in your project, then define your first eval and run checks to produce a report.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate AI code sessions and generate quantitative reliability metrics?▼

You can evaluate AI code sessions by using a structured harness to define, run, and assess tests. This framework generates pass@k and pass^k style reports to quantify AI reliability across code changes.

What is a regression eval framework for AI-assisted software engineering?▼

A regression eval framework safeguards against regressions across code changes by applying structured checks. It verifies AI outputs against predefined functional goals using deterministic capability and regression evaluations.

How do I set up a deterministic eval lifecycle for AI code generation?▼

You set up an eval lifecycle by organizing tests under a .claude/evals directory. This provides formal guidance for implementation, evaluation, and reporting to standardize AI behavior assessments.

Can I use both code and model graders to assess AI capability evaluations?▼

Yes, the framework supports both code and model graders. These graders assess AI-driven code sessions against predefined criteria to verify deterministic capability and regression evaluations.

What is the best way to standardize AI code reviews using automated testing?▼

The best way to standardize reviews is implementing an eval-driven development harness. It automates defining tests, running them, and recording results to guide AI behavior quantitatively across code changes.

Related Skills

Explore More Software Engineering Skills→
rongarederongarede
4

eval-harness

Evaluate Claude Code sessions with capability and regression evals.

Community
Advanced
LincyawLincyaw

Eval Harness Skill

Implement eval-driven development with code, model, and human graders.

Community
Advanced
rjroyrjroy
6

prep-plan

Create persistent implementation plans and save them as lore artifacts.

Community
Advanced
ryasrkryasrk

eval-harness

Formalize evaluation frameworks for AI code sessions with pass@k and pass^k metrics.

Community
Advanced