skills.rest
slowman2084
slowman2084COMMUNITY
Explore all skills by this author →
GitHub

meta-eval-judge

Score agent outputs against YAML rubrics or gold references with verified calculations.

6|Updated Mar 5, 2026
Search Tags:#evaluation#rubric#prompt-engineering#agent-testing#structured-report#automated-scoring#runlog-analysis
One-click install
npx skills add https://github.com/slowman2084/meta-agent --skill meta-eval-judge
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: meta-eval-judge
Source: https://github.com/slowman2084/meta-agent/tree/main/source/skills/meta-eval-judge
Command: npx skills add https://github.com/slowman2084/meta-agent --skill meta-eval-judge

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill automatically and rigorously evaluates a single agent or subagent output against a provided YAML rubric or a gold-reference ExpectedOutput, producing an evidence-backed numeric score, a validated calculation trace, and actionable suggestions for prompt engineering.

Core Features & Use Cases

  • Rubric-driven scoring: Parse atomic YAML rubrics, evaluate each criterion with direct evidence excerpts, sum achieved and positive points, and compute a mathematically verified final score.
  • Gold-reference comparison: When Judge is missing, compare ActualOutput to ExpectedOutput across semantic consistency, content completeness, and format/experience, with weighted aggregation.
  • Structured, machine-parseable reports: Emit a standardized Markdown evaluation report including total score, per-dimension judgments, full mathematical verification, advantages, shortcomings with root-cause tags, and specific prompt-level improvement suggestions.
  • RunLog-aware diagnostics: Optionally ingest run logs to assess tool-calling patterns, retries, and efficiency, and reflect those findings in relevant dimensions.
  • Use cases: CI test harness for agent development, subagent evaluation within meta-plan orchestration, and iterative prompt-engineering feedback loops.

Quick Start

Evaluate a single test case by supplying the TestCaseFile path, CaseIndex, and ActualOutputFile so the skill reads inputs, runs the chosen evaluation mode, and writes the structured case_N_eval_result.md to the specified output directory.

Frequently Asked Questions about meta-eval-judge

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate agent outputs against a YAML rubric automatically?▼

To evaluate agent outputs against a YAML rubric automatically, you supply the TestCaseFile path, CaseIndex, and ActualOutputFile. The skill parses atomic rubric criteria, extracts direct evidence excerpts, and calculates a mathematically verified final score.

What is the best way to compare agent outputs to a gold-reference expected output?▼

Comparing agent outputs to a gold-reference expected output involves evaluating semantic consistency, content completeness, and format. When a rubric is missing, the skill performs weighted aggregation across these dimensions to produce an evidence-backed numeric score.

Can I use run logs to diagnose agent tool-calling patterns during evaluation?▼

Yes, you can use run logs to diagnose agent tool-calling patterns during evaluation. The skill optionally ingests run logs to assess retries and efficiency, reflecting those diagnostic findings in the relevant evaluation dimensions of the structured report.

How do I generate structured evaluation reports for a CI test harness?▼

Generating structured evaluation reports for a CI test harness requires scoring single test-case outputs and writing the results to a specified directory. The skill emits a standardized Markdown report including per-dimension judgments, mathematical verification, and prompt-level improvement suggestions.

Does automated scoring provide prompt engineering suggestions for iterative improvement?▼

Yes, automated scoring provides prompt engineering suggestions for iterative improvement. The structured evaluation report identifies shortcomings with root-cause tags and offers specific, actionable recommendations to refine prompts for subagents and automated test suites.

Related Skills

Explore More Software Engineering Skills→
slowman2084slowman2084
6

meta-testcase-gen

Generate ten YAML test cases with empty ExpectedOutput and Judge fields.

Community
Advanced
slowman2084slowman2084
6

meta-retrospective

Analyze prompt histories to detect anti-patterns and generate retrospective reports.

Community
Advanced
jtvarisjtvaris

apple-intelligence

Integrate on-device Foundation Models with session management and structured outputs.

Community
Advanced
yashs33244yashs33244
1

investigate

Diagnose failures through systematic root-cause investigation across four phases.

Community
Advanced