skills.rest
aleonsa
aleonsaCOMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Define and execute capability and regression evaluations using pass@k and pass^k metrics.

1|Updated Mar 10, 2026
Search Tags:#testing#ai development#metrics#evaluation#regression testing#edd
One-click install
npx skills add https://github.com/aleonsa/claude-config --skill eval-harness-aleonsa
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/aleonsa/claude-config/tree/main/claude/skills/eval-harness
Command: npx skills add https://github.com/aleonsa/claude-config --skill eval-harness-aleonsa

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides a structured framework for evaluating AI agent performance and reliability, enabling a more rigorous and systematic approach to AI development and testing.

Core Features & Use Cases

  • Eval-Driven Development (EDD): Implements principles where evaluations define expected behavior before implementation.
  • Metrics & Reporting: Supports pass@k and pass^k metrics, generating detailed reports for capability and regression evals.
  • Use Case: When developing a new AI feature, use this Skill to define specific capability and regression tests upfront, then run these tests after implementation to ensure the feature works as expected and hasn't broken existing functionality.

Quick Start

Use the eval-harness skill to define a new capability evaluation for the 'user-authentication' feature.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is Eval-Driven Development and how does it formalize AI agent testing?▼

pass@k and pass^k are reliability metrics used to evaluate AI capability and regression. The framework generates detailed reports using these metrics to measure if new features work as expected and haven't broken existing functionality.

How do I set up capability and regression testing for a new AI feature?▼

The framework supports code-based, model-based, and human graders to evaluate AI workflows. This allows you to measure agent reliability using diverse grading mechanisms tailored to your specific evaluation criteria.

Can I use this framework for continuous integration and regression testing in AI workflows?▼

The framework supports code-based, model-based, and human graders to evaluate AI workflows. This allows you to measure agent reliability using diverse grading mechanisms tailored to your specific evaluation criteria.

What metrics are used to measure AI reliability during evaluation?▼

Yes, the framework facilitates continuous integration and regression testing for AI workflows. It executes capability and regression evaluations to systematically ensure new implementations do not break existing AI agent functionality.

What types of graders can I use to evaluate AI agent performance?▼

You define specific capability and regression evaluations upfront before implementation. After building the feature, you execute these evaluations to ensure the AI agent works as expected and hasn't introduced regressions.

Related Skills

Explore More Software Engineering Skills→
lgwanailgwanai
4

gsd-add-tests

Generate unit and E2E tests from SUMMARY, CONTEXT, and VERIFICATION specs.

Community
Advanced
VenTheZoneVenTheZone
1

eval-harness

Define capability and regression evaluations with code-based, model-based, and human graders.

Community
Advanced
daeha76daeha76
2

eval-harness

Create and execute capability and regression evaluations with code, model, or human grading.

Community
Advanced
LvcidPsycheLvcidPsyche
7

test-gen

Generate unit tests for functions, classes, and modules from source code.

Community
Intermediate