skills.rest
databrickslabs
databrickslabsOFFICIAL
Explore all skills by this author →
WebsiteGitHub

databricks-mlflow-evaluation

Evaluate GenAI agents with MLflow 3 GenAI evaluation APIs and built-in scorers.

38|12|Updated Apr 24, 2026
Search Tags:#prompt optimization#agent monitoring#mlflow#databricks#trace analysis#unity catalog#genai evaluation
One-click install
npx skills add https://github.com/databrickslabs/coding-agents-databricks-apps --skill databricks-mlflow-evaluation-databrickslabs
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/databrickslabs/coding-agents-databricks-apps/tree/main/.claude/skills/databricks-mlflow-evaluation
Command: npx skills add https://github.com/databrickslabs/coding-agents-databricks-apps --skill databricks-mlflow-evaluation-databrickslabs

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill solves the complexity of evaluating, monitoring, and improving GenAI agents built on Databricks. It eliminates guesswork by providing exact API patterns, common pitfalls, and end-to-end workflows for MLflow 3 GenAI evaluation, so developers can confidently measure agent quality, diagnose issues from traces, and systematically improve prompts using domain expert feedback.

Core Features & Use Cases

  • End-to-End Evaluation Workflows: Step-by-step guidance for first-time setup, production trace analysis, regression detection, custom scorer development, and performance optimization using the mlflow.genai.evaluate() API.
  • Production Monitoring & Trace Ingestion: Patterns for storing traces in Unity Catalog, instrumenting Databricks Apps and Model Serving endpoints, and running continuous quality monitoring with automated scorers.
  • Judge Alignment & Prompt Optimization: Workflows for aligning LLM judges with domain expert preferences using MemAlign, and automatically improving system prompts with GEPA and the optimize_prompts() function based on aligned judge feedback.

Quick Start

Use the databricks-mlflow-evaluation skill to run your first agent evaluation by importing your agent, defining a predict function, and calling mlflow.genai.evaluate() with built-in scorers like Safety and Guidelines against a small dataset of test inputs.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate GenAI agents using MLflow 3 on Databricks?▼

To evaluate GenAI agents using MLflow 3, import your agent, define a predict function, and call mlflow.genai.evaluate() with built-in scorers like Safety and Guidelines against a dataset of test inputs.

How do I store production traces from Databricks Model Serving in Unity Catalog?▼

Store production traces in Unity Catalog by instrumenting Databricks Apps and Model Serving endpoints, requiring mlflow[databricks]>=3.9.0 with explicit MODIFY and SELECT permissions on trace tables.

How do I align LLM judges with domain expert preferences for GenAI evaluation?▼

Align LLM judges with domain expert preferences using the MemAlign workflow, which systematically calibrates custom scorers to match expert feedback for accurate GenAI evaluation.

Can I automatically optimize system prompts for my Databricks GenAI agents?▼

Automatically optimize system prompts using the optimize_prompts() function with GEPA, which improves agent performance based on aligned judge feedback from domain expert evaluations.

What MLflow version is required for GenAI evaluation and trace ingestion?▼

GenAI evaluation requires mlflow[databricks]>=3.1.0, while storing production traces in Unity Catalog requires version >=3.9.0 to ensure proper trace table integration and monitoring.

How does trace analysis help monitor GenAI agent quality in production?▼

Trace analysis enables continuous quality monitoring by ingesting production traces into Unity Catalog, running automated scorers to detect regressions, and profiling agent performance over time.

Related Skills

Explore More Software Engineering Skills→
teegin-gteegin-g

databricks-model-serving

Deploy MLflow models and GenAI agents to Databricks model-serving endpoints.

Community
Advanced
aaronachermannaaronachermann

databricks-mlflow-evaluation

Evaluate GenAI agent outputs with MLflow GenAI evaluation APIs.

Community
Advanced
m19cm19c

databricks-model-serving

Deploy and query MLflow models and AI agents on Databricks Model Serving endpoints.

Community
Advanced
Zack2626-okZack2626-ok

databricks-mlflow-evaluation

Automates MLflow GenAI evaluation with built-in or custom scorers and GEPA prompt optimization.

Community
Advanced