skills.rest
Blackkadder
BlackkadderCOMMUNITY
Explore all skills by this author →
GitHub

databricks-mlflow-evaluation

Run MLflow GenAI evaluations with custom scorers, judges, and trace datasets.

Updated Mar 19, 2026
Search Tags:#evaluation#scorers#mlflow#unity-catalog#genai#gepa#judge-alignment
One-click install
npx skills add https://github.com/Blackkadder/databricks-apps-and-agents-workshop --skill databricks-mlflow-evaluation-blackkadder
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/Blackkadder/databricks-apps-and-agents-workshop/tree/main/.claude/skills/databricks-mlflow-evaluation
Command: npx skills add https://github.com/Blackkadder/databricks-apps-and-agents-workshop --skill databricks-mlflow-evaluation-blackkadder

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Provide clear, production-ready patterns and reference material to design, run, and maintain MLflow GenAI evaluations so teams can measure and improve agent and RAG quality without ad-hoc scripts or fragile workflows.

Core Features & Use Cases

  • Evaluation workflows: End-to-end patterns for first-time setup, building eval datasets from production traces, running mlflow.genai.evaluate, and comparing named runs for regression detection.
  • Scorers & Judges: How to use built-in scorers (Guidelines, Correctness, Safety, RetrievalGroundedness), author @scorer functions, and create custom LLM judges with make_judge.
  • Judge alignment & optimization: Procedures for aligning judges with MemAlign using SME labels and for automated prompt improvement with GEPA (optimize_prompts).
  • Trace & dataset patterns: Converting tagged traces into evaluation datasets, MLflow-managed UC datasets, and patterns for trace ingestion and production monitoring.
  • Operational best practices: Critical gotchas, API signatures, data schema requirements (nested inputs, predict_fn signature), and guidance for production monitoring and cost-aware embedding choices.

Quick Start

Run mlflow.genai.evaluate with a dataset of records containing an inputs key (and expectations when needed), a predict_fn that accepts unpacked kwargs, and scorers such as Safety(), Guidelines(name="x", guidelines="..."), or an aligned judge.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run MLflow GenAI evaluation on my agent traces?▼

Run MLflow GenAI evaluation by calling mlflow.genai.evaluate with a dataset of records containing an inputs key, a predict_fn accepting unpacked kwargs, and configured scorers like Safety or Guidelines.

How do I create custom LLM judges for MLflow GenAI evaluation?▼

Create custom LLM judges for MLflow GenAI evaluation by using the make_judge function or authoring @scorer functions to define specific evaluation criteria for your agents.

Can I align MLflow judges with subject matter expert feedback?▼

Align MLflow judges with subject matter expert feedback by using the MemAlign procedure with SME labels, ensuring your LLM judges accurately reflect human quality assessments.

What MLflow version is required for Unity Catalog trace ingestion?▼

Unity Catalog trace ingestion requires MLflow version 3.9 or higher, while running standard GenAI evaluations requires MLflow version 3.1 or higher.

How do I optimize prompts for MLflow GenAI scorers?▼

Optimize prompts for MLflow GenAI scorers by running the GEPA automated prompt improvement procedure using the optimize_prompts function to enhance evaluation quality.

What data schema is needed for MLflow GenAI evaluation datasets?▼

MLflow GenAI evaluation datasets require a nested inputs schema with records containing an inputs key, and expectations when needed, to properly pass arguments to the predict_fn.

Related Skills

Explore More Data & Analytics Skills→
aaronachermannaaronachermann

databricks-mlflow-evaluation

Evaluate GenAI agent outputs with MLflow GenAI evaluation APIs.

Community
Advanced
Zack2626-okZack2626-ok

databricks-mlflow-evaluation

Automates MLflow GenAI evaluation with built-in or custom scorers and GEPA prompt optimization.

Community
Advanced
Arize-aiArize-ai
11.0k

phoenix-evals

Build and run Phoenix evaluators for AI/LLM applications.

Official
Advanced
ken-cavanagh-gleanken-cavanagh-glean
4

evaluation

Design evaluation frameworks with rubrics and LLM-as-judge workflows.

Community
Advanced