skills.rest
LiamVDB1
LiamVDB1COMMUNITY
Explore all skills by this author →
GitHub

eval-harness

Define and run capability and regression evals with code-based, model-based, or human graders.

Updated Feb 13, 2026
Search Tags:#testing#development#ai#evaluation#framework#regression
One-click install
npx skills add https://github.com/LiamVDB1/opencode-config --skill eval-harness-liamvdb1
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/LiamVDB1/opencode-config/tree/main/skills/eval-harness
Command: npx skills add https://github.com/LiamVDB1/opencode-config --skill eval-harness-liamvdb1

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a structured framework for evaluating AI-generated code and outputs, enabling eval-driven development (EDD) principles to improve reliability and track regressions.

Core Features & Use Cases

  • Define Evals: Create capability and regression tests before implementation.
  • Automated Evaluation: Run code-based or model-based graders to assess AI performance.
  • Metrics Tracking: Monitor pass@k and pass^k metrics for reliability.
  • Use Case: When developing a new AI feature, define its expected behavior and success criteria using this Skill, then use it to continuously test and benchmark the AI's implementation against those criteria.

Quick Start

Use the eval-harness skill to define a new capability evaluation for user authentication.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
What is eval-driven development for AI-assisted coding?▼

You can evaluate AI-generated code by defining capability and regression tests using a formal evaluation framework. The framework supports code-based, model-based, and human graders to automatically assess AI performance and track reliability metrics like pass@k and pass^k.

How do I track AI regression testing metrics?▼

To set up an evaluation framework for AI capabilities, define your expected behavior and success criteria for the AI feature. Use the framework to create capability evaluations, then execute code-based or model-based graders to assess the AI's implementation against those predefined criteria.

Can I use model-based graders to evaluate AI outputs?▼

Using a formal evaluation framework for AI development allows you to define capability and regression tests upfront, automate evaluation with various graders, and track pass@k metrics. This structured approach improves reliability and ensures AI implementations meet predefined success criteria.

How do I define capability evaluations before AI implementation?▼

Regression testing in AI-assisted development is handled by defining regression tests and executing them through an evaluation framework. The framework runs graders to assess AI performance and tracks pass^k metrics, allowing developers to monitor reliability and detect regressions in AI-generated code.

Related Skills

Explore More Software Engineering Skills→
VjayRamVjayRam

dspy-expert

Define DSPy signatures and modules with deterministic evaluation across splits.

Community
Advanced
LincyawLincyaw

Eval Harness Skill

Implement eval-driven development with code, model, and human graders.

Community
Advanced
rongarederongarede
4

eval-harness

Evaluate Claude Code sessions with capability and regression evals.

Community
Advanced
VenTheZoneVenTheZone
1

eval-harness

Define capability and regression evaluations with code-based, model-based, and human graders.

Community
Advanced