skills.rest
AndyHsuTW
AndyHsuTWCOMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Implement eval-driven development with code-based, model-based, and human grading.

Updated Feb 21, 2026
Search Tags:#testing#ai development#evaluation#reliability#regression#edd
One-click install
npx skills add https://github.com/AndyHsuTW/everything-llm-workspace --skill eval-harness-andyhsutw
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/AndyHsuTW/everything-llm-workspace/tree/main/.agents/skills/eval-harness
Command: npx skills add https://github.com/AndyHsuTW/everything-llm-workspace --skill eval-harness-andyhsutw

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides a structured approach to developing AI agents by treating evaluations as first-class citizens, ensuring reliability and preventing regressions.

Core Features & Use Cases

  • Eval-Driven Development (EDD): Define expected behavior before implementation.
  • Automated Testing: Run capability and regression evals to catch issues early.
  • Metrics Tracking: Monitor pass@k and pass^k metrics for reliability.
  • Use Case: Before adding a new feature to your AI agent, define its expected outputs and behaviors using this Skill. Then, implement the feature and run the evals to confirm it works as intended and hasn't broken existing functionality.

Quick Start

Use the eval-harness skill to define a new capability evaluation for the 'user-authentication' feature.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is eval-driven development for AI agents?▼

Eval-driven development (EDD) treats formal evaluations as first-class citizens, requiring you to define expected behavior and outputs before implementation to ensure AI reliability and prevent regressions.

How do I prevent regressions when adding features to an AI agent?▼

Run automated capability and regression evals after implementing new features. This catches issues early by verifying the new functionality works as intended and hasn't broken existing behavior.

Can I use both model-based and human grading for AI capability evaluations?▼

Yes, this formal evaluation framework supports code-based, model-based, and human grading for capability and regression evals, allowing flexible assessment strategies for AI development.

What metrics should I track to measure AI agent reliability?▼

Track pass@k and pass^k metrics to monitor AI agent reliability. These metrics quantify formal evaluation results and help ensure consistent performance during development.

Does this AI evaluation framework require any external dependencies?▼

No, this eval-driven development framework operates without external dependencies, providing a standalone harness to define expected behaviors, run evals, and track reliability metrics.

When do I need to run formal regression testing for AI development?▼

Run formal regression testing whenever you add a new feature to your AI agent. Define expected outputs beforehand, implement the feature, then run evals to confirm functionality and prevent regressions.

Related Skills

Explore More Software Engineering Skills→
The-Skyy-Rose-Collection-LLCThe-Skyy-Rose-Collection-LLC
3

eval-harness

Define, run, and report evaluations with /eval commands.

Official
Intermediate
LincyawLincyaw

Eval Harness Skill

Implement eval-driven development with code, model, and human graders.

Community
Advanced
Clean1inesClean1ines

eval-harness

Evaluate Claude Code sessions with configurable graders and run pipelines.

Community
Advanced
ysyecustysyecust
40

eval-harness

Codify and automate Claude Code session evaluations with pass@k metrics.

Community
Intermediate