evals-creator

What problem does it solve?

This Skill eliminates the manual, error-prone effort of writing and keeping eval definitions in sync with evolving intents and skill contracts, by generating structured step and scenario evals from the source documents.

Core Features & Use Cases

• Step eval generation: Creates step evals mapped to intent failure conditions and constraint-driven artifact-verifiable rules, each with concrete, observable pass/fail criteria.
• Scenario eval generation: Builds E2E scenario evals from expectation success scenarios (or legacy intent scenarios) so play-level outcomes remain consistent and traceable.
• Traceable, binary checks: Ensures every eval is traceable to its source (failure condition, constraint, or scenario) and grounded in observable artifact state.
• Coverage reporting: Produces a coverage summary indicating which failure conditions, constraints, and scenarios are covered or uncovered.

Quick Start

Use evals-creator to generate updated step and scenario eval YAML by providing the intent_path, skill_contracts, output_path, and (optionally) expectation_path.