skills.rest
DigiStrique-Solutions
DigiStrique-SolutionsOFFICIAL
Explore all skills by this author →
GitHub

eval-harness

Automate AI prompt and output evaluation against ground-truth fixtures.

1|Updated Apr 2, 2026
Search Tags:#fixtures#prompts#regression-testing#evaluate#ground-truth
One-click install
npx skills add https://github.com/DigiStrique-Solutions/sentinel --skill eval-harness-digistrique-solutions
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-harness
Source: https://github.com/DigiStrique-Solutions/sentinel/tree/main/skills/eval-harness
Command: npx skills add https://github.com/DigiStrique-Solutions/sentinel --skill eval-harness-digistrique-solutions

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Evaluation harness for testing AI agent prompts and outputs against ground truth. Activates when designing eval scenarios, running prompt evaluations, interpreting results to decide whether a prompt change is an improvement. Setting up regression testing for agent outputs. Iterating on prompts that consistently fail verification.

Core Features & Use Cases

  • Eval Scenario Design: A structured test case for an AI agent. It includes a prompt, context, expectations, and ground truth.
  • Expectation Definition: Types of expectations (structural, content, quality, behavioral) with precise criteria.
  • Ground Truth Generation: Run agent, capture output, verify, and save verified fixtures for future comparisons.
  • Running Evaluations: Modes Generate, Regression, Full; execution flow and verification against expectations.
  • Iterative Prompt Improvement: Eval-Fix loop, targeted prompt refinements and re-evaluation.
  • Scenario Maintenance: Adding new scenarios and retiring outdated ones to maintain coverage.

Quick Start

Define a scenario, run the agent against it, and verify the output against the ground-truth fixture.

Frequently Asked Questions about eval-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run regression testing on AI prompts against ground truth fixtures?▼

Regression testing on AI prompts involves comparing agent outputs against saved ground-truth fixtures to detect regressions. You run an evaluation in Regression mode to verify whether prompt changes maintain expected behaviors and structural consistency.

What are the best ways to evaluate AI agent outputs for prompt iteration?▼

Evaluating AI agent outputs for prompt iteration uses an eval-fix loop: define a scenario with expectations, run the evaluation, interpret results, and apply targeted prompt refinements. Re-evaluating after each fix confirms whether the change is an improvement.

How do I generate ground truth fixtures for testing AI prompts?▼

Generating ground truth fixtures requires running the AI agent against a defined scenario, capturing the output, verifying it meets expectations, and saving the verified output. These fixtures become the baseline for future regression testing.

What types of expectations can I define for AI prompt evaluation scenarios?▼

Expectations for AI prompt evaluation scenarios include structural, content, quality, and behavioral criteria. These precise expectations define how the agent's output is verified against ground truth during the evaluation process.

How do I maintain test scenario coverage when iterating on AI prompts?▼

Maintaining test scenario coverage requires adding new scenarios to cover edge cases discovered during iteration and retiring outdated ones that no longer reflect the agent's intended behavior, ensuring the evaluation suite stays relevant.

Related Skills

Explore More Software Engineering Skills→
whichguywhichguy

review-fix-bench

Benchmark code-reviewer agent prompts with parallel test runs and F1 metrics.

Community
Advanced
getsentrygetsentry
905

prompt-optimizer

Optimize agent prompts across OpenAI, Claude, and Gemini with eval-driven refinements.

Official
Advanced
sfc-gh-dflipposfc-gh-dflippo
34

dbt Artifacts Package

Track dbt run history and execution metadata with the dbt_artifacts package.

Community
Intermediate
hkcm91hkcm91

triggering-ai-reflection

Trigger AI reflection cycles to evaluate generated content and refine prompts.

Community
Advanced