eval-dataset

Generates evaluation test cases from eval.md and eval.yaml for /eval-run workflows.

36|39|Updated Apr 2, 2026

Search Tags:#automation #validation #benchmark #dataset #evaluate #test-case-generation

Ready to run

npx skills add https://github.com/opendatahub-io/agent-eval-harness --skill eval-dataset

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-dataset
Source: https://github.com/opendatahub-io/agent-eval-harness/tree/main/skills/eval-dataset
Command: npx skills add https://github.com/opendatahub-io/agent-eval-harness --skill eval-dataset

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Automates the generation of end-to-end evaluation test cases for skills, ensuring coverage against a defined eval.yaml/spec and judge criteria.

Core Features & Use Cases

Generate diverse, judge-driven test cases aligned to dataset schemas and evaluation config.
Create context artifacts like inputs.yaml, annotations.yaml, answers.yaml, and companion files as needed.
Validate case structure and provide guidance for expanding or refining evaluation coverage.

Quick Start

Run /eval-dataset to generate a complete set of evaluation test cases for a target skill.

Frequently Asked Questions about eval-dataset

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I generate evaluation test cases for a dataset schema?▼

You generate evaluation test cases by parsing eval.md and eval.yaml configurations. This produces input.yaml, annotations.yaml, and optional answers.yaml artifacts aligned to your dataset schema and judge criteria.

What is automated test case generation for skill evaluation?▼

Automated test case generation creates diverse, judge-driven test cases aligned to your dataset schema and evaluation config. It validates case structure and provides guidance for expanding or refining evaluation coverage.

Do I need eval.yaml and eval.md files to build a validation dataset?▼

Yes, you need eval.yaml and eval.md files to build a validation dataset. The generator parses these configuration files to align with your dataset schema and judge criteria, producing the required input.yaml and annotations.yaml artifacts.

Can I validate dataset structure and expand evaluation coverage simultaneously?▼

Yes, you can validate dataset structure and expand coverage simultaneously. The generator applies validation checks defined by judges while creating context artifacts, providing guidance for refining or expanding your evaluation coverage.

What's the best way to automate benchmark dataset generation across multiple skills?▼

The best way to automate benchmark dataset generation across multiple skills is using a configuration-driven approach that parses eval.yaml to produce standardized input.yaml and annotations.yaml artifacts for /eval-run workflows.

Why does my generated annotations.yaml not match the judge criteria?▼

Your annotations.yaml may not match judge criteria if your eval.yaml configuration is misaligned with the dataset schema. The generator validates case structure against these defined judges to prevent misalignment.