eval-harness

Formalize evaluation frameworks for AI code sessions with pass@k and pass^k metrics.

Updated Mar 12, 2026

Search Tags:#testing #ai development #metrics #evaluation #reliability #regression testing

Ready to run

npx skills add https://github.com/ryasrk/AgentBrokeTheMatrix-CopilotVersion --skill eval-harness-ryasrk

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/ryasrk/AgentBrokeTheMatrix-CopilotVersion/tree/main/.github/skills/eval-harness
Command: npx skills add https://github.com/ryasrk/AgentBrokeTheMatrix-CopilotVersion --skill eval-harness-ryasrk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides a structured framework for evaluating AI model performance and reliability, enabling a more rigorous and repeatable development process for AI-assisted workflows.

Core Features & Use Cases

Eval-Driven Development (EDD): Implement unit-test-like evaluations for AI development.
Metric Tracking: Measure reliability using pass@k and pass^k metrics.
Use Case: When developing a new AI feature, define expected outcomes and success criteria upfront using this Skill, then continuously run evaluations to ensure the AI meets these standards and to catch regressions.

Quick Start

Define a new capability evaluation for a feature named 'user-authentication'.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is eval-driven development for AI features?▼

You can measure AI reliability using pass@k and pass^k metrics, which track performance across multiple attempts to ensure consistent outcomes and catch regressions during AI feature development.

How do I set up capability and regression evals for AI code sessions?▼

The framework supports code-based, model-based, and human grading methods for AI evaluation, allowing you to implement unit-test-like evaluations tailored to your specific reliability and regression testing requirements.

Can I use this framework to catch regressions in AI-assisted workflows?▼

Yes, you can catch regressions by continuously running capability and regression evals against your predefined success criteria, ensuring your AI-assisted workflows maintain reliability throughout development iterations.

How do I track pass@k and pass^k metrics for AI reliability?▼

You track pass@k and pass^k metrics within the evaluation framework to measure AI model reliability, providing quantitative tracking to ensure your code sessions consistently meet the expected performance standards.