eval-runner-designer

What problem does it solve?

This Skill closes the design gap between having eval cases on disk and having a real, trustworthy harness that can execute them, score them, and report results without overstating what happened.

Core Features & Use Cases

• Execution Semantics: Defines how each corpus case type runs, including fresh-session isolation, trigger checks, refusal-case handling, and discrimination scoring for expected versus excluded skills.
• Judging and Reporting: Separates deterministic checks from rubric-guided semantic judging, assigns honest states such as PASS, FAIL, ERROR, JUDGE-ERROR, QUARANTINED, and UNRUN, and keeps invalid or incomplete runs visible.
• Cost and Reliability Controls: Adds sampling tiers, spend caps, kill-switch behavior, flake policy, and advisory-first CI wiring so eval execution can be planned without laundering results.
• Use Case: A team with structural-only eval files uses this Skill to specify exactly how a future runner should behave before any code is built.

Quick Start

Use the eval-runner-designer skill to draft a complete, honest specification for how the repository’s skill evals would be executed, judged, reported, and budgeted.