skills.rest
itsadijmbt
itsadijmbtCOMMUNITY
Explore all skills by this author →
GitHub

databricks-mlflow-evaluation

Run MLflow GenAI evaluations with dataset-driven scoring and built-in scorers.

1|Updated Apr 24, 2026
Search Tags:#prompt optimization#unity catalog#llm scoring#trace ingestion#judge alignment#mlflow genai evaluation
One-click install
npx skills add https://github.com/itsadijmbt/SecureMCP-Servers --skill databricks-mlflow-evaluation-itsadijmbt
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/itsadijmbt/SecureMCP-Servers/tree/main/TEST_SERVERS/PORTED_TO_SECUREMCP/databrickslab-mcp/ai-dev-kit/databricks-skills/databricks-mlflow-evaluation
Command: npx skills add https://github.com/itsadijmbt/SecureMCP-Servers --skill databricks-mlflow-evaluation-itsadijmbt

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you reliably evaluate and monitor MLflow 3 GenAI agents so you can measure quality, catch regressions, and improve prompts using trustworthy scoring signals.

Core Features & Use Cases

  • MLflow 3 GenAI evaluation workflows: Run mlflow.genai.evaluate() with built-in scorers like Guidelines, Correctness, Safety, and RetrievalGroundedness.
  • Trace-to-dataset pipelines: Convert Unity Catalog stored production traces into evaluation datasets for repeatable testing.
  • Judge alignment and prompt optimization: Align a judge with MemAlign from SME feedback and optimize prompts with optimize_prompts() using GEPA for automated improvements.

Quick Start

Use the databricks-mlflow-evaluation skill to score your agent with mlflow.genai.evaluate() using Guidelines plus Correctness on a dataset that includes inputs and expectations.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate GenAI agents using MLflow 3?▼

You can evaluate GenAI agents by running mlflow.genai.evaluate() with built-in scorers like Guidelines, Correctness, Safety, and RetrievalGroundedness on a dataset containing inputs, optional outputs, and expectations to generate actionable quality metrics.

How do I convert Unity Catalog production traces into MLflow evaluation datasets?▼

You can convert Unity Catalog stored production traces into evaluation datasets using trace-to-dataset pipelines, enabling repeatable testing and regression detection across different agent versions.

Can I optimize LLM prompts automatically using MLflow GenAI evaluation?▼

Yes, you can optimize LLM prompts automatically using the optimize_prompts() function with GEPA, and align your judge model with MemAlign by incorporating subject matter expert feedback for trustworthy scoring signals.

What data schema is required for mlflow.genai.evaluate() to run successfully?▼

The mlflow.genai.evaluate() function requires nested dataset records containing inputs, optional outputs and expectations, scorer and judge compatibility with required schemas, and correct predict_fn kwargs unpacking to ensure accurate scoring.

Why am I getting schema compatibility errors during MLflow GenAI agent scoring?▼

MLflow GenAI agent scoring schema errors typically occur when dataset records lack required inputs or expectations, when judge schemas mismatch, or when predict_fn kwargs are not unpacked correctly during the evaluation run.

Related Skills

Explore More Software Engineering Skills→
ice-droid-99ice-droid-99

databricks-mlflow-evaluation

Set up MLflow 3 GenAI agent evaluations with mlflow.genai.evaluate().

Community
Advanced
kinnerkarmanishkinnerkarmanish

error-analysis

Analyze LLM traces to categorize failures and identify recurring patterns.

Community
Advanced
ThomazRossitoThomazRossito
31

databricks-mlflow-evaluation

Orchestrate MLflow GenAI evaluation pipelines for scoring and monitoring agent behavior.

Community
Advanced
BlackkadderBlackkadder

agent-evaluation

Evaluate and optimize LLM agent output quality using MLflow evaluation APIs.

Community
Advanced