databricks-mlflow-evaluation

Evaluate Generative AI agents and LLM applications with MLflow.

Updated Mar 10, 2026

Search Tags:#prompt optimization #llm #evaluation #agent testing #mlflow #genai #trace analysis

Ready to run

npx skills add https://github.com/slysik/databricks-claude-coding --skill databricks-mlflow-evaluation-slysik

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/slysik/databricks-claude-coding/tree/main/.agents/skills/databricks-mlflow-evaluation
Command: npx skills add https://github.com/slysik/databricks-claude-coding --skill databricks-mlflow-evaluation-slysik

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill streamlines the evaluation of Generative AI agents and LLM applications, enabling users to write, debug, and optimize their GenAI code effectively using MLflow.

Core Features & Use Cases

Automated Evaluation: Run mlflow.genai.evaluate() for deterministic testing.
Scorer Integration: Use built-in scorers (Guidelines, Correctness, Safety, RetrievalGroundedness) or create custom ones.
Trace Analysis: Debug agent behavior by analyzing execution traces.
Prompt Optimization: Improve prompts using GEPA with optimize_prompts().
Use Case: You've built a RAG agent and want to ensure its responses are relevant, grounded in retrieved documents, and adhere to safety guidelines. This Skill provides the tools to set up an evaluation run, analyze the results, and iterate on prompt improvements.

Quick Start

Use the databricks-mlflow-evaluation skill to evaluate your agent using the provided evaluation dataset and safety and guidelines scorers.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate Generative AI agents using MLflow?▼

MLflow enables GenAI agent evaluation through the mlflow.genai.evaluate() function, which tests agents against datasets using built-in scorers for guidelines, correctness, safety, and retrieval groundedness.

Can I create custom scorers for LLM application testing in MLflow?▼

Yes, MLflow supports custom scorer integration alongside built-in scorers like Guidelines, Correctness, Safety, and RetrievalGroundedness, allowing tailored LLM application evaluation metrics.

What is the best way to debug GenAI agent behavior during evaluation?▼

Trace analysis via mlflow.search_traces() is the best way to debug GenAI agent behavior, allowing you to inspect execution traces and identify issues in agent logic.

How do I optimize prompts for LLM applications with MLflow?▼

You can optimize LLM application prompts using GEPA via mlflow.genai.optimize_prompts(), which iteratively improves prompt performance based on evaluation results.

Does MLflow support evaluation for RAG agents?▼

Yes, MLflow supports RAG agent evaluation by providing RetrievalGroundedness scorers to ensure responses are grounded in retrieved documents and adhere to safety guidelines.

What data schema is needed for running mlflow.genai.evaluate()?▼

Running mlflow.genai.evaluate() requires an evaluation dataset conforming to specific data schemas, with detailed guidance provided on schemas and scorer interfaces.