skills.rest
aufrank
aufrankCOMMUNITY
Explore all skills by this author →
GitHub

eval-harness-kit

Build and execute deterministic evaluation suites for agent workflows using JSON manifests.

Updated Aug 27, 2026
Search Tags:#testing#metrics#agent#evaluation#regression#harness
One-click install
npx skills add https://github.com/aufrank/agent-skills --skill eval-harness-kit
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness-kit
Source: https://github.com/aufrank/agent-skills/tree/main/skills/eval-harness-kit
Command: npx skills add https://github.com/aufrank/agent-skills --skill eval-harness-kit

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill automates the creation and execution of evaluation suites for agent workflows, ensuring reproducible results and tracking capabilities or regressions.

Core Features & Use Cases

  • Manifest-driven Evals: Define tasks, inputs, and grading criteria in a JSON manifest.
  • Deterministic & LLM Grading: Supports exact match, regex, JSON comparison, and optional LLM rubrics.
  • Use Case: You've developed a new agent for summarizing documents. Use this Skill to create an evaluation suite with various documents and expected summary qualities, then run the agent against it to measure performance and identify regressions over time.

Quick Start

Run the example manifest using python <CODEX_HOME>/skills/eval-harness-kit/scripts/run_eval.py --manifest <path> --run-id <id>.

Frequently Asked Questions about eval-harness-kit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build deterministic evaluation suites for agent workflows?▼

To build deterministic evaluation suites, define tasks, inputs, outputs, and grading criteria in a JSON manifest. The harness then executes these suites against agent workflows to generate reproducible results and track regressions over time.

What grading methods can I use for agent evaluation?▼

Agent evaluation supports exact match, regex, JSON comparison, and LLM rubrics. These grading criteria are specified within JSON manifests to deterministically assess agent outputs for single-turn and agentic task execution.

How do I track regressions in agent performance automatically?▼

You can track regressions by running agent workflows against an evaluation suite. The harness generates JSONL logs and summary reports for each run, enabling direct comparison of capability tracking and performance metrics over time.

Can I evaluate single-turn and multi-step agentic tasks using JSON manifests?▼

Yes, JSON manifests support defining tasks for both single-turn and agentic execution. You specify the inputs, expected outputs, and grading criteria directly in the manifest before running the evaluation suite.

How do I run an evaluation suite from the command line?▼

Run an evaluation suite by executing the Python script with your manifest path and a run identifier. The command `python <CODEX_HOME>/skills/eval-harness-kit/scripts/run_eval.py --manifest <path> --run-id <id>` processes the manifest and outputs logs.

Related Skills

Explore More Software Engineering Skills→
VjayRamVjayRam

dspy-expert

Define DSPy signatures and modules with deterministic evaluation across splits.

Community
Advanced
rysweetrysweet
2

learning-path-builder

Create personalized learning paths with sequenced resources and progress milestones.

Community
Advanced
ZaibunisZaibunis

spec-kit-agent-orchestrator

Generate workflow prompts and scripts for multi-agent orchestration using Speckit methodology.

Community
Advanced
lucasgaldinoslucasgaldinos

agentic-eval

Automate agent output evaluation with rubric-based scoring and self-critique.

Community
Advanced