What problem does it solve? Once you have written a judge prompt for an AI agent, you need evidence that it actually distinguishes between actions it should allow, block, revise, or escalate. This Skill produces a structured evaluation suite of realistic test cases so you can measure judge accuracy before production incidents reveal the gaps. ## Core Features & Use Cases - Four-outcome test coverage: Generates at least 20 test cases distributed across ALLOW, BLOCK, REVISE, and ESCALATE outcomes, each with a full action proposal, expected outcome, reasoning, and the consequence of a wrong decision. - Boundary-failure focus: Designs cases around mundane, realistic failures such as weak evidence behind confident justifications and authorization scope creep, not just adversarial scenarios. - Metrics guidance: Recommends tracking false allow rate, false block rate, escalation rate, revision rate, and per-criterion performance to detect when the judge needs tuning. - Use Case: After drafting a judge prompt that reviews outbound emails, use this Skill to produce a test suite covering unauthorized recipients, missing evidence, and ambiguous policy cases, then run it to measure the judge's false allow rate. ## Quick Start Use the judge-evaluation-suite-generator skill to create an evaluation suite for my judge that reviews outbound email actions against authorization, evidence, risk, and policy criteria.
