skills.rest
ninyawee
ninyaweeCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

eval-harness

Define capability and regression evals for AI code sessions using pass@k metrics.

Updated Aug 27, 2026
Search Tags:#testing#ai development#metrics#evaluation#regression#edd
One-click install
npx skills add https://github.com/ninyawee/dotfiles --skill eval-harness-ninyawee
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/ninyawee/dotfiles/tree/main/dot_claude/skills/eval-harness
Command: npx skills add https://github.com/ninyawee/dotfiles --skill eval-harness-ninyawee

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill provides a structured framework for evaluating AI-generated code and outputs, ensuring reliability and tracking regressions through formal evaluation processes.

Core Features & Use Cases

  • Formal Evaluation: Define clear pass/fail criteria for AI tasks using capability and regression evals.
  • Metric Tracking: Measure performance using pass@k and pass^k metrics for reliability assessment.
  • Use Case: When developing a new AI feature, use this Skill to define expected outcomes and continuously test the AI's implementation against these criteria, ensuring it meets quality standards before deployment.

Quick Start

Use the eval-harness skill to define a new capability evaluation for the 'user-authentication' feature.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is eval-driven development for AI and how does it track regressions?▼

Eval-driven development (EDD) is a formal framework for evaluating AI code sessions. It tracks regressions and measures performance using pass@k and pass^k metrics to ensure AI outputs maintain reliability.

How do I define pass/fail criteria for AI code generation evaluations?▼

You define pass/fail criteria for AI code generation by setting up capability and regression evals. This structured evaluation process tests the AI's implementation against expected outcomes to ensure it meets quality standards.

When do I need formal capability and regression evals in my AI workflow?▼

You need formal capability and regression evals when developing new AI features to continuously test implementations. This ensures the AI meets defined quality standards and prevents performance degradation before deployment.

Can I use pass@k and pass^k metrics to measure AI code session reliability?▼

Yes, you can measure AI code session reliability using pass@k and pass^k metrics. These metrics facilitate structured reporting and assess the performance of AI-generated code within the evaluation framework.

What's the best way to structure reporting for AI development workflows?▼

The best way to structure reporting for AI workflows is implementing a formal evaluation harness. It tracks performance metrics like pass@k and pass^k, enabling structured assessment of AI-generated code and feature reliability.

Does the eval-harness skill require external dependencies to evaluate AI outputs?▼

No external dependencies are required. The eval-harness skill operates independently using internal scripts and references to implement the evaluation framework for AI-generated code and track performance.

Related Skills

Explore More Software Engineering Skills→
galeepgaleep

caveman-stats

Analyze Claude Code session logs for actual token usage and savings.

Community
Basic
LincyawLincyaw

Eval Harness Skill

Implement eval-driven development with code, model, and human graders.

Community
Advanced
xu-xiangxu-xiang
1.8k

eval-harness

Define, execute, and report capability and regression evals for Claude Code sessions.

Community
Intermediate
aman-2709aman-2709
12

eval-harness

Define, run, and report Claude Code session evaluations with pass/fail criteria.

Community
Advanced