skills.rest
ThomazRossito
ThomazRossitoCOMMUNITY
Explore all skills by this author →
Website𝕏@thomazrossitoGitHub

databricks-mlflow-evaluation

Orchestrate MLflow GenAI evaluation pipelines for scoring and monitoring agent behavior.

31|18|Updated Apr 4, 2026
Search Tags:#evaluation#scorers#mlflow#prompt-optimization#genai#trace-ingestion#judge-alignment
One-click install
npx skills add https://github.com/ThomazRossito/data-agents --skill databricks-mlflow-evaluation-thomazrossito
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/ThomazRossito/data-agents/tree/main/skills/databricks/databricks-mlflow-evaluation
Command: npx skills add https://github.com/ThomazRossito/data-agents --skill databricks-mlflow-evaluation-thomazrossito

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Provides a complete reference and operational playbook for running reliable MLflow GenAI evaluations, building eval datasets from traces, aligning LLM judges with domain experts, and enabling production monitoring so teams can measure and improve agent quality without guesswork.

Core Features & Use Cases

  • End-to-end evaluation workflows: Step-by-step patterns for initial setup, running evaluations, comparing runs, and detecting regressions.
  • Scorers & judges: Guidance on built-in scorers (Guidelines, Correctness, Safety, RetrievalGroundedness), custom scorer patterns, and make_judge + judge alignment with MemAlign.
  • Trace-to-dataset and UC ingestion: Instructions for converting production traces into evaluation datasets, linking Unity Catalog schemas, and configuring production monitoring.
  • Prompt optimization: Patterns for building optimization datasets (inputs + expectations) and running optimize_prompts() with GEPA driven by aligned judges.
  • Critical gotchas & APIs: CRITICAL-interfaces and GOTCHAS guidance to avoid common failures (data schema, predict_fn signature, trace setup, label schema naming).

Quick Start

Run an evaluation with mlflow.genai.evaluate() using a nested eval dataset, a predict function that accepts unpacked kwargs, and the desired scorers to generate traces and metrics.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run MLflow GenAI evaluation on my agent?▼

To run MLflow GenAI evaluation, use mlflow.genai.evaluate() with a nested eval dataset, a predict function accepting unpacked kwargs, and desired scorers to generate traces and metrics.

What built-in scorers are available for MLflow GenAI evaluation?▼

MLflow GenAI evaluation supports built-in scorers including Guidelines, Correctness, Safety, and RetrievalGroundedness to measure agent behavior across various quality dimensions.

How do I build an evaluation dataset from production traces in Unity Catalog?▼

Building an evaluation dataset from production traces involves converting traces into evaluation datasets, linking Unity Catalog schemas, and configuring production monitoring to measure agent quality.

Can I align LLM judges with SME feedback before running prompt optimization?▼

Yes, you can align LLM judges with SME feedback using make_judge and MemAlign judge alignment before building optimization datasets and running optimize_prompts() with GEPA.

What are common gotchas when configuring MLflow GenAI evaluation pipelines?▼

Common MLflow GenAI evaluation gotchas involve data schema issues, predict_fn signature mismatches, trace setup errors, and label schema naming conventions that cause pipeline failures.

What is GEPA prompt optimization and how does it work with MLflow?▼

GEPA prompt optimization uses aligned judges to run optimize_prompts() on datasets containing inputs and expectations, refining agent prompts to improve overall evaluation metrics.

Related Skills

Explore More Data & Analytics Skills→
BlackkadderBlackkadder

databricks-mlflow-evaluation

Run MLflow GenAI evaluations with custom scorers, judges, and trace datasets.

Community
Advanced
mirakuimirakui

databricks-mlflow-evaluation

Configure Databricks MLflow GenAI evaluation workflows with mlflow.genai.evaluate().

Community
Advanced
Zack2626-okZack2626-ok

databricks-mlflow-evaluation

Automates MLflow GenAI evaluation with built-in or custom scorers and GEPA prompt optimization.

Community
Advanced
raphaelmansuyraphaelmansuy
2

agent-evaluation-mlflow

Implement agent evaluation and safety gates using MLflow 3.x scoring and tracing workflows.

Community
Advanced