eval-harness

Define, execute, and report capability and regression evaluations using pass@k and pass^k metrics.

Updated Feb 17, 2026

Search Tags:#code quality #testing #ai development #metrics #evaluation #regression testing

Ready to run

npx skills add https://github.com/Gatsbyhateyou/gatsby-website --skill eval-harness-gatsbyhateyou

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/Gatsbyhateyou/gatsby-website/tree/main/.agents/skills/eval-harness
Command: npx skills add https://github.com/Gatsbyhateyou/gatsby-website --skill eval-harness-gatsbyhateyou

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides a structured framework for evaluating AI-generated code and outputs, ensuring reliability and quality through formal testing and metrics.

Core Features & Use Cases

Eval-Driven Development (EDD): Implement a development process where evaluations (tests) are defined before coding.
Automated Testing: Define and run capability and regression evals to measure AI performance.
Metrics Tracking: Utilize metrics like pass@k and pass^k to quantify reliability.
Use Case: When developing a new feature with an AI assistant, define the expected behavior and success criteria upfront using this Skill. Then, run the defined evals to verify the AI's implementation and track its performance over time.

Quick Start

Use the eval-harness skill to define a new capability evaluation for the 'user-authentication' feature.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

What is eval-driven development and how does it measure AI reliability?▼

Eval-driven development defines evaluations before coding to ensure AI-assisted development reliability. It measures performance using pass@k and pass^k metrics to quantify AI-generated code quality through formal capability and regression testing.

How do I set up regression testing for AI-generated code?▼

To set up regression testing for AI-generated code, define capability evaluations specifying expected behavior and success criteria upfront. Execute these evals to verify the AI's implementation and track its performance over time.

Can I use both automated and human grading mechanisms for AI evaluation?▼

Yes, AI evaluation supports code-based, model-based, and human grading mechanisms. This comprehensive assessment approach allows you to combine automated metrics tracking with manual review for evaluating AI-generated outputs.

What's the best way to track AI code quality metrics during development?▼

The best way to track AI code quality metrics is by implementing a formal evaluation framework. It facilitates defining, executing, and reporting capability and regression evaluations to continuously measure AI reliability using pass@k and pass^k metrics.

Does formal AI evaluation work without external dependencies?▼

Yes, formal AI evaluation works without external dependencies. The framework implements eval-driven development principles natively, allowing you to define and run capability evaluations independently using code-based, model-based, and human grading mechanisms.