ai-evaluation-harness

What problem does it solve?

This Skill prevents silent quality or safety regressions in AI features by turning model behavior into a measurable, versioned evaluation process with explicit pass/fail thresholds.

Core Features & Use Cases

• Versioned Eval Datasets: Builds representative, adversarial/red-team, and regression cases into one maintained dataset so changes are tested against real usage and known failures.
• Dimension-Specific Grading: Uses deterministic checks, reference comparisons, and pinned judges to score task quality, schema adherence, safety/refusal, groundedness, injection resistance, latency, and cost.
• Security-Focused Gates: Encodes injection, jailbreak, exfiltration, tool-misuse, and disclosure cases as hard safety gates that block unsafe model or prompt changes.
• Use Case: A team updating a chatbot prompt can run this Skill to prove the new version still follows the output schema, refuses unsafe requests, and does not leak secrets or follow injected instructions.

Quick Start

Use the ai-evaluation-harness skill to design a versioned eval suite for our AI feature, define pass thresholds, and set up a manual CI gate that blocks regressions.