eval-mlflow

Logs MLflow evaluation runs and syncs datasets bidirectionally with feedback.

36|39|Updated Apr 2, 2026

Search Tags:#automation #feedback #evaluation #traces #tracking #dataset #mlflow

Ready to run

npx skills add https://github.com/opendatahub-io/agent-eval-harness --skill eval-mlflow

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-mlflow
Source: https://github.com/opendatahub-io/agent-eval-harness/tree/main/skills/eval-mlflow
Command: npx skills add https://github.com/opendatahub-io/agent-eval-harness --skill eval-mlflow

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires mlflow, pyyaml, and includes scripts (resource) components.

What problem does it solve?

Bridges evaluation data flow by logging run results and syncing datasets with MLflow, enabling end-to-end tracking of experiments and feedback between the harness and MLflow traces.

Core Features & Use Cases

MLflow experiment tracking: log parameters, metrics, and artifacts from evaluation runs to MLflow for consistent analytics.
Dataset synchronization: push evaluation cases to MLflow datasets and maintain cross-run traceability.
Feedback integration: attach judge and human feedback to traces and pull annotations back into the evaluation pipeline for optimization.
Use case: a data science team validates model performance across many runs and compares results in a single MLflow dashboard.

Quick Start

Install the skill and configure MLflow tracking, then run the evaluation workflow to start logging results and syncing data.

Frequently Asked Questions about eval-mlflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I log evaluation run results and metrics to MLflow?▼

You can log evaluation run results to MLflow by using this skill to bridge data flow, automatically tracking parameters, metrics, and artifacts for consistent analytics across local and remote setups.

Can I sync evaluation datasets with MLflow to maintain traceability?▼

Yes, you can sync evaluation datasets with MLflow to push evaluation cases and maintain cross-run traceability from raw cases to final assessments, ensuring deterministic tracking across your pipeline.

What is the best way to attach judge and human feedback to MLflow traces?▼

The best way to attach feedback to MLflow traces is using this skill's bidirectional feedback integration, which attaches annotations to traces and pulls them back into the evaluation pipeline for optimization.

Does MLflow evaluation tracking work with both local and remote setups?▼

Yes, MLflow evaluation tracking works across both local and remote MLflow setups, allowing you to maintain end-to-end traceability and deterministic tracking regardless of your infrastructure configuration.

How do I pull annotations from MLflow back into my evaluation pipeline?▼

You can pull annotations back into your evaluation pipeline by applying this skill's bidirectional feedback mechanism, which integrates judge and human feedback from MLflow traces directly into your harness for optimization.