evals-creator

To generate scenario evals from intent YAML, provide the intent_path, skill_contracts, output_path, and optionally expectation_path to produce structured, traceable step and scenario eval definitions mapped to failure conditions.

Artifact verification in agentic workflow evals ensures every generated check is grounded in observable artifact state, enforcing binary, traceable pass/fail criteria mapped directly to specific constraints and failure conditions.

Keep eval definitions in sync by generating structured step and scenario eval YAML directly from source intent and skill contracts, eliminating manual effort and ensuring checks remain traceable to evolving constraints.

Yes, you can generate coverage reports. The process produces a coverage summary indicating which failure conditions, constraints, and scenarios are currently covered or uncovered by the generated eval definitions.

You do not need an expectation.yaml file to generate step evals. The process generates step evals from intent.yaml and skill contracts, using expectation.yaml optionally to build E2E scenario evals.

Map failure conditions to observable binary checks by parsing constraints and failure_conditions from source documents, generating step evals that enforce artifact-verifiable constraint handling with concrete pass/fail criteria.