skills.rest
leary-poken
leary-pokenCOMMUNITY
Explore all skills by this author →
GitHub

databricks-mlflow-evaluation

Run MLflow GenAI evaluate() to score agent outputs with configurable scorers.

1|Updated Mar 17, 2026
Search Tags:#evaluation#alignment#traces#scorers#mlflow#datasets#genai
One-click install
npx skills add https://github.com/leary-poken/ai-dev-kit --skill databricks-mlflow-evaluation-leary-poken
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/leary-poken/ai-dev-kit/tree/main/databricks-skills/databricks-mlflow-evaluation
Command: npx skills add https://github.com/leary-poken/ai-dev-kit --skill databricks-mlflow-evaluation-leary-poken

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

MLflow GenAI Evaluation analyzes agent outputs against domain-specific standards using MLflow GenAI workflows.

Core Features & Use Cases

  • End-to-end evaluation patterns including dataset creation, evaluation execution, judge alignment, and GEPA-based prompt optimization to drive measurable quality signals.
  • Production monitoring capabilities to consistently score production traces with configurable scorers and samples.
  • Support for MLflow-managed datasets and Unity Catalog trace ingestion for auditable, reproducible evaluation data and regression testing.
  • Tools for comparing runs, identifying regressions, and driving iterative improvements to prompts and judges.

Quick Start

Install the MLflow 3 GenAI Evaluation tool in your Databricks project and run mlflow.genai.evaluate() with your eval data and scorers to start.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate GenAI agent outputs against domain-specific standards in MLflow?▼

To evaluate GenAI agent outputs in MLflow, configure an MLflow environment, define a base judge, provide custom scorers, and run mlflow.genai.evaluate() with your datasets to generate actionable quality signals and diagnostics.

Can I use MLflow managed datasets and Unity Catalog for trace ingestion during evaluation?▼

Yes, MLflow GenAI Evaluation supports MLflow-managed datasets and Unity Catalog trace ingestion, enabling auditable, reproducible evaluation data for regression testing and consistent production monitoring.

What is GEPA-based prompt optimization and does it work with MLflow evaluation?▼

GEPA-based prompt optimization is an end-to-end evaluation pattern within MLflow GenAI workflows that iteratively improves prompts and judges to drive measurable quality signals from agent outputs.

How do I compare evaluation runs and identify regressions in GenAI agents?▼

You can compare evaluation runs and identify regressions using built-in MLflow tools that analyze scoring results, highlight quality drops, and drive iterative improvements to prompts and judges.

What is the best way to consistently score production traces with MLflow?▼

The best way to consistently score production traces is using MLflow GenAI Evaluation's production monitoring capabilities, which apply configurable scorers and samples to ongoing trace data.

Related Skills

Explore More Data & Analytics Skills→
ice-droid-99ice-droid-99

agent-evaluation

Evaluate and improve LLM agent outputs using MLflow GenAI workflows.

Community
Advanced
coval-aicoval-ai
2

review-llm-annotations-and-improve-prompt

Analyze human versus LLM judge annotation discrepancies to refine evaluation prompts.

Official
Advanced
ThomazRossitoThomazRossito
31

databricks-mlflow-evaluation

Orchestrate MLflow GenAI evaluation pipelines for scoring and monitoring agent behavior.

Community
Advanced
datasciencemonkeydatasciencemonkey
27

databricks-mlflow-evaluation

Automate MLflow GenAI evaluation workflows for Databricks agents with structured datasets and reference files.

Community
Advanced