skills.rest
opendatahub-io
opendatahub-ioOFFICIAL
Explore all skills by this author →
WebsiteGitHub

eval-mlflow

Logs MLflow evaluation runs and syncs datasets bidirectionally with feedback.

36|39|Updated Apr 2, 2026
Search Tags:#automation#feedback#evaluation#traces#tracking#dataset#mlflow
One-click install
npx skills add https://github.com/opendatahub-io/agent-eval-harness --skill eval-mlflow
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-mlflow
Source: https://github.com/opendatahub-io/agent-eval-harness/tree/main/skills/eval-mlflow
Command: npx skills add https://github.com/opendatahub-io/agent-eval-harness --skill eval-mlflow

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires mlflow, pyyaml, and includes scripts (resource) components.

What problem does it solve?

Bridges evaluation data flow by logging run results and syncing datasets with MLflow, enabling end-to-end tracking of experiments and feedback between the harness and MLflow traces.

Core Features & Use Cases

  • MLflow experiment tracking: log parameters, metrics, and artifacts from evaluation runs to MLflow for consistent analytics.
  • Dataset synchronization: push evaluation cases to MLflow datasets and maintain cross-run traceability.
  • Feedback integration: attach judge and human feedback to traces and pull annotations back into the evaluation pipeline for optimization.
  • Use case: a data science team validates model performance across many runs and compares results in a single MLflow dashboard.

Quick Start

Install the skill and configure MLflow tracking, then run the evaluation workflow to start logging results and syncing data.

Frequently Asked Questions about eval-mlflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I log evaluation run results and metrics to MLflow?▼

You can log evaluation run results to MLflow by using this skill to bridge data flow, automatically tracking parameters, metrics, and artifacts for consistent analytics across local and remote setups.

Can I sync evaluation datasets with MLflow to maintain traceability?▼

Yes, you can sync evaluation datasets with MLflow to push evaluation cases and maintain cross-run traceability from raw cases to final assessments, ensuring deterministic tracking across your pipeline.

What is the best way to attach judge and human feedback to MLflow traces?▼

The best way to attach feedback to MLflow traces is using this skill's bidirectional feedback integration, which attaches annotations to traces and pulls them back into the evaluation pipeline for optimization.

Does MLflow evaluation tracking work with both local and remote setups?▼

Yes, MLflow evaluation tracking works across both local and remote MLflow setups, allowing you to maintain end-to-end traceability and deterministic tracking regardless of your infrastructure configuration.

How do I pull annotations from MLflow back into my evaluation pipeline?▼

You can pull annotations back into your evaluation pipeline by applying this skill's bidirectional feedback mechanism, which integrates judge and human feedback from MLflow traces directly into your harness for optimization.

Related Skills

Explore More Data & Analytics Skills→
terrylicaterrylica
62

mlflow-python

Log MLflow metrics, parameters, and artifacts using the Python API.

Community
Advanced
thbraetthbraet

experiment-compare

Compare MLflow experiment runs across metrics, hyperparameters, tags, and artifacts.

Community
Intermediate
fgarofalo56fgarofalo56
6

mlflow

Manage the ML lifecycle with MLflow for experiment tracking and model deployment.

Community
Advanced
Albatross679Albatross679

mlflow-monitor

Launch the MLflow UI to monitor and compare training experiment metrics.

Community
Intermediate