skills.rest
opendatahub-io
opendatahub-ioOFFICIAL
Explore all skills by this author →
WebsiteGitHub

eval-review

Analyze evaluation results to identify alignment gaps between automated checks and human assessment.

36|39|Updated Apr 2, 2026
Search Tags:#evaluation#transcripts#mlflow#skill-improvement#human-feedback#case-analysis#review-results
One-click install
npx skills add https://github.com/opendatahub-io/agent-eval-harness --skill eval-review
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-review
Source: https://github.com/opendatahub-io/agent-eval-harness/tree/main/skills/eval-review
Command: npx skills add https://github.com/opendatahub-io/agent-eval-harness --skill eval-review

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Facilitates thorough, human-in-the-loop review of evaluation results by surfacing judge outcomes, qualitative feedback, and actionable improvement opportunities for evaluation SKILLs.

Core Features & Use Cases

  • Analyze per-case judge scores and outputs to identify where automated checks align with or miss human expectations.
  • Collect and synthesize user feedback and transcripts to detect recurring issues across runs, guiding SKILL.md improvements.
  • Propose targeted SKILL.md changes and evaluation workflow adjustments to drive faster, iterative improvements.

Quick Start

Review a run by providing --run-id and follow the prompts to collect human feedback and generate improvement recommendations.

Frequently Asked Questions about eval-review

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I analyze MLflow evaluation results to find gaps between automated scores and human feedback?▼

To analyze MLflow evaluation results, aggregate per-case judge scores, outputs, and transcripts to identify where automated checks miss human expectations. This surfaces recurring issues across runs, enabling structured recommendations to improve your evaluation flow.

What is human-in-the-loop evaluation review and when do I need it for skill improvement?▼

Human-in-the-loop evaluation review is the process of surfacing judge outcomes and qualitative feedback to validate automated checks. You need it when automated scores misalign with human expectations, requiring targeted adjustments to drive iterative skill improvements.

How do I generate SKILL.md diffs from per-case judge scores and user transcripts?▼

Generate SKILL.md diffs by collecting human feedback and transcripts for a specific run, synthesizing recurring patterns, and producing structured recommendations. Provide the run-id to start the review process and follow prompts to guide targeted improvements.

Can I use evaluation transcripts to detect recurring issues across multiple MLflow runs?▼

Yes, evaluation transcripts are collected and synthesized alongside user feedback to detect recurring issues across multiple runs. Aggregating this data surfaces recurring patterns, guiding targeted changes to SKILL.md and the evaluation workflow.

What's the best way to improve evaluation workflows when automated checks disagree with human assessment?▼

The best way to resolve disagreements is analyzing per-case judge scores and outputs to pinpoint alignment gaps. Synthesize qualitative feedback and transcripts to propose targeted workflow adjustments, driving iterative enhancements to the evaluation process.

Do I need a specific run-id to start reviewing evaluation results and collecting human feedback?▼

Yes, you need a run-id to start reviewing evaluation results. Provide the run-id via the command line to initiate the review process, which collects human feedback and generates structured improvement recommendations for your evaluation flow.

Related Skills

Explore More Data & Analytics Skills→
leary-pokenleary-poken
1

databricks-mlflow-evaluation

Run MLflow GenAI evaluate() to score agent outputs with configurable scorers.

Community
Advanced
ScotterMonkScotterMonk
37

skill-creator

Create, test, and evaluate AI skills with SKILL.md files.

Community
Advanced
ccfccf

llm-evaluation

Automate LLM evaluation with automated metrics, human feedback, and benchmarking.

Community
Advanced
amurataamurata
6

llm-evaluation

Evaluate LLM applications using automated metrics, human feedback, and benchmarking.

Community
Advanced