opendatahub-ioOfficial·7 Agent Skills Included

agent-eval-harness

Measure, score, and improve skill performance with evals

Evaluates skills and agent capabilities through a single declarative eval.yaml covering analysis, test case generation, execution, judging, and reporting. Eliminates guesswork about skill quality by scoring outputs with code checks and LLM judges, tracking cost, and detecting regressions across runs. Logs traces and datasets to MLflow, compares models with ANOVA statistics, and auto-fixes failing skills through an optimization loop.

36|39

↗

npx skills add opendatahub-io/agent-eval-harness --all -g -y

Available:

Explains the harness execution model, architecture, and skill layout so the agent knows how to configure, run, and extend evaluations correctly.

All Skills in This Repository (7)

Pure Emerald Level Indicators

📦 In Repo

opendatahub-io

eval-analyze

Analyze a target skill to generate eval.yaml configuration for /eval-run.

eval-review

Analyze evaluation results to identify alignment gaps between automated checks and human assessment.

eval-setup

Configure and validate the agent-eval-harness environment with MLflow tracking and API key checks.

eval-run

Execute skills against test cases, score outputs with judges, and generate run summaries.

eval-mlflow

Logs MLflow evaluation runs and syncs datasets bidirectionally with feedback.

eval-optimize

Analyze eval results and edit SKILL.md to fix judge failures.

eval-dataset

Generates evaluation test cases from eval.md and eval.yaml for /eval-run workflows.

Official

Advanced

Frequently Asked Questions

FAQPage Schema

How to install agent-eval-harness?▼

Run `npx skills add opendatahub-io/agent-eval-harness --all -g -y` in your terminal to install all evaluation skills globally.

How to test if my skill actually works?▼

Run /eval-analyze to generate an eval.yaml from your skill, then /eval-dataset to create test cases and /eval-run to execute and score them with judges.

Can it compare different models on the same eval?▼

Yes. The eval-anova skill runs factorial experiments across models and effort levels, then applies repeated-measures ANOVA and cost/quality Pareto analysis to show which differences are statistically significant.

Does it work with MLflow?▼

Yes. The eval-mlflow skill syncs datasets, logs run metrics and artifacts, and attaches judge and human feedback to MLflow traces.

Can it improve a failing skill automatically?▼

Yes. The eval-optimize skill reads judge failures and transcripts, edits the skill's instructions, re-runs the eval, and checks for regressions in an automated loop.