skills.rest
deneb-cygnus-dev
deneb-cygnus-devCOMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Formalize AI-assisted development workflow evaluations with EDD principles and pass@k metrics.

Updated Aug 23, 2026
Search Tags:#testing#ai development#metrics#evaluation#regression testing#edd
One-click install
npx skills add https://github.com/deneb-cygnus-dev/dot-agent --skill eval-harness-deneb-cygnus-dev
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/deneb-cygnus-dev/dot-agent/tree/main/skills/eval-harness
Command: npx skills add https://github.com/deneb-cygnus-dev/dot-agent --skill eval-harness-deneb-cygnus-dev

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a structured approach to evaluating AI-driven development, ensuring reliability and preventing regressions in AI-assisted workflows.

Core Features & Use Cases

  • Eval-Driven Development (EDD): Implements principles where evaluations act as unit tests for AI development.
  • Metrics: Supports pass@k and pass^k metrics for measuring reliability.
  • Use Case: When developing a new AI feature, define capability and regression evals upfront to ensure the feature works as expected and doesn't break existing functionality.

Quick Start

Use the eval-harness skill to define a new capability evaluation for the 'add-authentication' feature.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is eval-driven development for AI workflows?▼

Eval-driven development (EDD) formalizes AI evaluation by treating evaluations as unit tests for AI development, ensuring reliability and preventing regressions in AI-assisted workflows.

How do I measure AI agent reliability using pass@k and pass^k metrics?▼

You can measure AI agent reliability by applying pass@k and pass^k metrics within a formal evaluation harness to benchmark performance and track regressions across prompt changes.

How do I set up regression testing for AI prompt changes?▼

Define capability and regression evaluations upfront using an eval harness to structured-test prompt changes, ensuring new AI features work as expected without breaking existing functionality.

Can I benchmark AI agent performance for capability evaluations?▼

Yes, you can benchmark AI agent performance by defining structured capability evaluations that measure reliability and detect regressions during AI-assisted development workflows.

When should I use a formal evaluation framework for AI development?▼

Use a formal evaluation framework when developing new AI features to define capability and regression evals upfront, ensuring the feature works as expected and preventing workflow regressions.

Related Skills

Explore More Software Engineering Skills→
understudylabsunderstudylabs
10

capture-evidence

Generate harness, metric, splits, and baseline artifacts from workloads.

Official
Advanced
sayasaya8039sayasaya8039
2

eval-harness

Define pass/fail criteria and run regression evals for Claude Code agents.

Community
Advanced
khetansarveshkhetansarvesh
1

eval-harness

Define and run structured evals for Claude Code sessions with pass@k metrics.

Community
Advanced
kk20300113-pngkk20300113-png

eval-harness

Define pass/fail criteria and calculate pass@k metrics for Claude Code agent evals.

Community
Advanced