eval-analyze

Analyze a target skill to generate eval.yaml configuration for /eval-run.

36|39|Updated Apr 2, 2026

Search Tags:#automation #validation #yaml #skill #analyze #datasets #eval

Ready to run

npx skills add https://github.com/opendatahub-io/agent-eval-harness --skill eval-analyze

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-analyze
Source: https://github.com/opendatahub-io/agent-eval-harness/tree/main/skills/eval-analyze
Command: npx skills add https://github.com/opendatahub-io/agent-eval-harness --skill eval-analyze

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill automates the creation of eval.yaml for the agent-eval harness by analyzing a target skill's SKILL.md, sub-skills, prompts, scripts, and test cases to derive a complete evaluation configuration.

Core Features & Use Cases

Recursive skill analysis: reads the target SKILL.md and any sub-skills, along with prompts, scripts, and test cases, to derive a complete evaluation configuration.
Grounded configuration: ensures dataset schema, outputs, judges, models, and thresholds are anchored in observed files rather than placeholders.
Workflow automation: produces a ready-to-run eval.yaml and caches eval.md for future reference.
Trigger integration: can be invoked to prepare evaluation infrastructure when eval.yaml is missing.

Quick Start

Invoke /eval-analyze against a target skill to auto-generate a complete eval.yaml for evaluation.

Frequently Asked Questions about eval-analyze

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I auto-generate an eval.yaml configuration for a custom skill?▼

To auto-generate an eval.yaml configuration, invoke the eval-analyze skill against your target skill. It recursively reads SKILL.md, sub-skills, prompts, and scripts to derive a grounded evaluation configuration without placeholders.

What is the best way to analyze skill datasets and derive evaluation thresholds?▼

The best way to analyze skill datasets and derive evaluation thresholds is using recursive skill analysis. It reads your target skill's test cases and files to ensure dataset schema, outputs, and judges are anchored in observed data.

How does eval.yaml generation work for the agent-eval harness?▼

Eval.yaml generation works by deeply reading your target skill directory, including sub-skills and scripts. It produces the exact configuration required by the /eval-run harness, caching an eval.md file for future reference.

Can I use eval-analyze to prepare evaluation infrastructure when eval.yaml is missing?▼

Yes, you can use eval-analyze to prepare evaluation infrastructure when eval.yaml is missing. It serves as a trigger integration that automatically analyzes your skill files to output the needed configuration.

Do I need to manually define dataset schema and judges for my skill evaluation?▼

You do not need to manually define dataset schema and judges. The tool ensures every field, including models and thresholds, is automatically grounded in the actual files observed in your skill directory.

What limitations exist when deriving configurations for skills with unstructured prompts?▼

A key limitation is that configuration generation depends on reading actual files in the skill directory. If prompts, scripts, and test cases are missing or unstructured, it cannot ground the dataset schema and thresholds accurately.