databricks-mlflow-evaluation

Run MLflow GenAI evaluations with custom scorers and analyze traces.

Updated Mar 12, 2026

Search Tags:#prompt optimization #llm #evaluation #agent testing #mlflow #genai #trace analysis

Ready to run

npx skills add https://github.com/juanlamadrid20/coda --skill databricks-mlflow-evaluation-juanlamadrid20

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/juanlamadrid20/coda/tree/main/.claude/skills/databricks-mlflow-evaluation
Command: npx skills add https://github.com/juanlamadrid20/coda --skill databricks-mlflow-evaluation-juanlamadrid20

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you rigorously evaluate your GenAI agents and LLM applications, identify regressions, and systematically improve their quality using MLflow's advanced evaluation tools.

Core Features & Use Cases

Run Evaluations: Execute mlflow.genai.evaluate() with custom scorers and datasets.
Analyze Traces: Debug agent behavior and performance using detailed trace analysis.
Optimize Prompts: Automatically improve prompts using GEPA with aligned judges.
Production Monitoring: Set up continuous evaluation of live traffic.
Use Case: You've deployed a RAG agent and want to ensure its responses are always grounded in retrieved documents and adhere to brand guidelines. Use this Skill to set up an evaluation pipeline that runs mlflow.genai.evaluate() with RetrievalGroundedness and custom Guidelines scorers, analyze failing traces, and iterate on prompt improvements.

Quick Start

Use the databricks-mlflow-evaluation skill to run a baseline evaluation on your agent using the provided sample dataset and safety and guidelines scorers.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate GenAI agents and LLM applications using MLflow?▼

To evaluate GenAI agents using MLflow, you execute the `mlflow.genai.evaluate()` function with custom scorers and datasets. This process allows you to systematically measure agent quality, identify regressions, and analyze detailed execution traces for debugging.

What is trace analysis for LLM agents and how does it help debug performance?▼

Trace analysis for LLM agents involves examining detailed execution traces captured during evaluation to understand internal behavior. It helps developers debug performance issues by revealing exactly where failures occur within complex generative AI workflows.

Can I automatically optimize prompts for LLMs using MLflow?▼

Yes, you can automatically optimize prompts for LLMs using MLflow by leveraging GEPA with aligned judges. This feature systematically improves prompt quality by iteratively refining instructions based on evaluation feedback.

How do I set up continuous production monitoring for live GenAI traffic?▼

You can set up continuous production monitoring for live GenAI traffic by configuring an evaluation pipeline using MLflow. This approach continuously assesses live responses using built-in checks like Safety and Correctness to maintain quality standards.

Does MLflow evaluation support custom scorers for domain-specific quality assessment?▼

Yes, MLflow evaluation supports custom scorers and advanced features like judge alignment with MemAlign for domain-specific quality assessment. You can define custom guidelines to ensure responses adhere to specific brand or retrieval requirements.

What is the best way to ensure RAG agent responses are grounded in retrieved documents?▼

The best way to ensure RAG agent responses are grounded is to run `mlflow.genai.evaluate()` with the `RetrievalGroundedness` scorer. This validates that outputs remain factually tied to your retrieved documents and adhere to guidelines.