eval-dataset

You generate evaluation test cases by parsing eval.md and eval.yaml configurations. This produces input.yaml, annotations.yaml, and optional answers.yaml artifacts aligned to your dataset schema and judge criteria.

Automated test case generation creates diverse, judge-driven test cases aligned to your dataset schema and evaluation config. It validates case structure and provides guidance for expanding or refining evaluation coverage.

Yes, you need eval.yaml and eval.md files to build a validation dataset. The generator parses these configuration files to align with your dataset schema and judge criteria, producing the required input.yaml and annotations.yaml artifacts.

Yes, you can validate dataset structure and expand coverage simultaneously. The generator applies validation checks defined by judges while creating context artifacts, providing guidance for refining or expanding your evaluation coverage.

The best way to automate benchmark dataset generation across multiple skills is using a configuration-driven approach that parses eval.yaml to produce standardized input.yaml and annotations.yaml artifacts for /eval-run workflows.

Your annotations.yaml may not match judge criteria if your eval.yaml configuration is misaligned with the dataset schema. The generator validates case structure against these defined judges to prevent misalignment.