skills.rest
ryasrk
ryasrkCOMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Formalize evaluation frameworks for AI code sessions with pass@k and pass^k metrics.

Updated Mar 12, 2026
Search Tags:#testing#ai development#metrics#evaluation#reliability#regression testing
One-click install
npx skills add https://github.com/ryasrk/AgentBrokeTheMatrix-CopilotVersion --skill eval-harness-ryasrk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/ryasrk/AgentBrokeTheMatrix-CopilotVersion/tree/main/.github/skills/eval-harness
Command: npx skills add https://github.com/ryasrk/AgentBrokeTheMatrix-CopilotVersion --skill eval-harness-ryasrk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides a structured framework for evaluating AI model performance and reliability, enabling a more rigorous and repeatable development process for AI-assisted workflows.

Core Features & Use Cases

  • Eval-Driven Development (EDD): Implement unit-test-like evaluations for AI development.
  • Metric Tracking: Measure reliability using pass@k and pass^k metrics.
  • Use Case: When developing a new AI feature, define expected outcomes and success criteria upfront using this Skill, then continuously run evaluations to ensure the AI meets these standards and to catch regressions.

Quick Start

Define a new capability evaluation for a feature named 'user-authentication'.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is eval-driven development for AI features?▼

You can measure AI reliability using pass@k and pass^k metrics, which track performance across multiple attempts to ensure consistent outcomes and catch regressions during AI feature development.

How do I set up capability and regression evals for AI code sessions?▼

The framework supports code-based, model-based, and human grading methods for AI evaluation, allowing you to implement unit-test-like evaluations tailored to your specific reliability and regression testing requirements.

Can I use this framework to catch regressions in AI-assisted workflows?▼

Yes, you can catch regressions by continuously running capability and regression evals against your predefined success criteria, ensuring your AI-assisted workflows maintain reliability throughout development iterations.

How do I track pass@k and pass^k metrics for AI reliability?▼

You track pass@k and pass^k metrics within the evaluation framework to measure AI model reliability, providing quantitative tracking to ensure your code sessions consistently meet the expected performance standards.

Related Skills

Explore More Software Engineering Skills→
cescraflicescrafli

eval-harness

Automate Claude Code session evaluations with code, model, and human graders.

Community
Advanced
LincyawLincyaw

Eval Harness Skill

Implement eval-driven development with code, model, and human graders.

Community
Advanced
aleonsaaleonsa

eval-harness

Define and execute capability and regression evaluations using pass@k and pass^k metrics.

Community
Advanced
jikimejikime
5

jikime-workflow-eval

Define and execute capability, regression, and quality evals with automated grading.

Community
Advanced