eval-harness

Construct evaluation loops grading AI agent output with an LLM judge.

746|130|Updated Jun 30, 2026

Search Tags:#quality control #model comparison #regression testing #AI evaluation #prompt testing

Ready to run

npx skills add https://github.com/Archive228/loopkit --skill eval-harness-archive228

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/Archive228/loopkit/tree/main/skills/eval-harness
Command: npx skills add https://github.com/Archive228/loopkit --skill eval-harness-archive228

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill enables the creation of repeatable evaluation loops for AI agents, ensuring consistent quality control for prompt and skill changes.

Core Features & Use Cases

Repeatable Evaluation: Build eval loops that grade agent output with an LLM judge, ensuring changes are scored against a baseline.
Three-Stage Loop: Includes stages for inputs, runner, and verifier, providing a structured approach to evaluation.
Use Case: Ideal for tuning prompts, changing skills, comparing models, regression testing, and ensuring the effectiveness of AI workflows.

Quick Start

Run the eval-harness skill by providing inputs and executing the loop.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate regression testing for AI prompts and skills?▼

Automate regression testing for AI prompts by constructing a repeatable evaluation loop that grades agent output using an LLM judge, ensuring changes are consistently scored against a baseline.

What is an LLM judge evaluation loop for AI agents?▼

An LLM judge evaluation loop is a structured process for grading AI agent outputs across three stages: inputs, runner, and verifier, maintaining baseline consistency for quality control.

How do I set up model comparison testing for AI workflows?▼

Set up model comparison testing by running a structured evaluation loop that processes inputs, executes the runner, and verifies outputs to evaluate and compare different AI models.

Can I use an AI evaluation harness for prompt tuning and workflow verification?▼

Yes, an AI evaluation harness is designed specifically for tuning prompts and verifying workflows by grading agent outputs against a baseline to ensure effectiveness and consistent quality.

Do I need any external dependencies to run AI agent evaluation loops?▼

No external dependencies are required to run AI agent evaluation loops, as the harness operates independently using its built-in scripts and references to grade outputs and maintain baseline consistency.

What are the limitations of using an LLM judge for AI quality control?▼

Using an LLM judge for AI quality control limits evaluations to the judge model's grading accuracy, requiring structured inputs and clear baselines to effectively verify workflow changes.