eval-harness

Implement eval-driven development frameworks with code-based, model-based, and human grading.

5|Updated Feb 6, 2026

Search Tags:#testing #ai development #metrics #evaluation #regression testing #edd

Ready to run

npx skills add https://github.com/FaisalAlqarni/sp-ecc --skill eval-harness-faisalalqarni

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/FaisalAlqarni/sp-ecc/tree/main/skills/eval-harness
Command: npx skills add https://github.com/FaisalAlqarni/sp-ecc --skill eval-harness-faisalalqarni

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides a structured framework for evaluating AI model performance, ensuring reliability and tracking regressions through formal evaluation processes.

Core Features & Use Cases

Eval-Driven Development (EDD): Treats AI evaluations as unit tests, defining expected behavior before implementation.
Capability Evals: Test new AI functionalities.
Regression Evals: Ensure existing functionalities are not broken by changes.
Multiple Grader Types: Supports code-based, model-based, and human grading.
Metrics: Tracks pass@k and pass^k for reliability measurement.
Use Case: A team developing a new AI feature can use this harness to define success criteria, implement the feature, and then run automated evaluations to confirm it meets the requirements and doesn't break existing capabilities.

Quick Start

Use the eval-harness skill to define a new capability evaluation for the 'user-authentication' feature.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is eval-driven development and how does it apply to AI testing?▼

Eval-driven development (EDD) treats AI evaluations as unit tests, defining expected model behavior before implementation begins. This approach ensures structured AI development by establishing success criteria upfront and continuously monitoring for regressions.

How do I set up capability and regression evaluations for AI features?▼

You can set up capability evaluations to test new AI functionalities and regression evaluations to ensure changes don't break existing features. The harness supports code-based, model-based, and human grading to measure pass@k and pass^k metrics.

What metrics are used to track AI model reliability during regression testing?▼

Regression testing tracks AI model reliability using pass@k and pass^k metrics. These metrics measure consistency and reliability, helping teams monitor whether existing capabilities are broken by recent code changes.

Can I use automated grading for AI capability evaluations?▼

Yes, capability evaluations support automated grading through code-based and model-based graders. This allows teams to automatically confirm new AI features meet defined requirements without relying solely on human grading.

When should I use model-based grading versus human grading in AI evaluations?▼

Model-based grading automates the evaluation of AI capabilities for faster regression testing, while human grading provides subjective assessment for complex behaviors. The harness supports both, along with code-based grading, for flexible evaluation.

Does formal AI evaluation work without external dependencies?▼

Yes, this formal evaluation framework operates without external dependencies, allowing teams to implement eval-driven development principles and track pass@k metrics directly within their AI development sessions.