skills.rest
ndhananj
ndhananjCOMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Implement Evaluation-Driven Development with code, model, and human scorers.

1|Updated Feb 7, 2026
Search Tags:#code quality#testing#devops#ai development#evaluation#regression testing
One-click install
npx skills add https://github.com/ndhananj/codex-agent-setup --skill eval-harness-ndhananj
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/ndhananj/codex-agent-setup/tree/main/docs/zh-CN/skills/eval-harness
Command: npx skills add https://github.com/ndhananj/codex-agent-setup --skill eval-harness-ndhananj

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill addresses the challenge of reliably evaluating AI-generated code by implementing a formal, structured evaluation framework, promoting a disciplined approach to AI development.

Core Features & Use Cases

  • Evaluation-Driven Development (EDD): Integrates evaluation as a core part of the development lifecycle, treating evaluations as "unit tests for AI."
  • Diverse Evaluation Types: Supports both capability assessments (testing new functionalities) and regression assessments (ensuring existing functionalities remain intact).
  • Multiple Scorer Types: Accommodates code-based (deterministic), model-based (AI-assisted), and human-in-the-loop scoring.
  • Key Metrics: Tracks crucial metrics like pass@k and pass^k to quantify reliability and success rates.
  • Use Case: Before deploying a new AI-generated feature, use this Skill to define and run a suite of tests that verify its functionality, ensure it doesn't break existing code, and measure its success against predefined metrics.

Quick Start

Use the eval-harness skill to define a new capability evaluation for user authentication.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate AI-generated code for reliability before deployment?▼

You can evaluate AI-generated code by implementing Evaluation-Driven Development (EDD) principles. This framework treats evaluations as unit tests for AI, allowing you to define and run suites that verify new functionality, track pass@k and pass^k metrics, and measure success against predefined metrics before deployment.

What is Evaluation-Driven Development for AI coding sessions?▼

Evaluation-Driven Development (EDD) integrates evaluation as a core part of the development lifecycle for AI coding sessions. It treats evaluations as unit tests for AI, supporting capability assessments for new functionalities and regression assessments to ensure existing code remains intact.

Can I use both automated and human-in-the-loop scoring for AI code regression testing?▼

Yes, you can use code-based (deterministic), model-based (AI-assisted), and human-in-the-loop scorers for AI code regression testing. This accommodates diverse evaluation types to ensure existing functionalities remain intact during development.

How does this framework track AI code capability and regression metrics?▼

The framework tracks AI code capability and regression metrics by quantifying reliability and success rates using pass@k and pass^k metrics. It includes defined workflows for evaluation definition, execution, reporting, and integrated storage for evaluation artifacts.

Does the eval-harness skill support model-based and human scorers for capability evaluations?▼

Yes, the eval-harness skill supports model-based, code-based, and human scorers for capability evaluations. This allows you to test new AI-generated functionalities using deterministic, AI-assisted, or human-in-the-loop scoring methods.

Related Skills

Explore More Software Engineering Skills→
ThrookieThrookie
2

eval-harness

Define, implement, and run capability and regression evaluations for Claude Code sessions.

Community
Advanced
RooseveltAdvisorsRooseveltAdvisors
1

Evals

Evaluate AI agent workflows with code, model, and human graders.

Community
Advanced
barrydobsonbarrydobson

devops

Advocate DevOps principles including CI/CD, IaC, automated testing, monitoring, and security scanning for delivery pipelines.

Community
Intermediate
aleonsaaleonsa

eval-harness

Define and execute capability and regression evaluations using pass@k and pass^k metrics.

Community
Advanced