skills.rest
slysik
slysikCOMMUNITY
Explore all skills by this author →
GitHub

databricks-mlflow-evaluation

Evaluate Generative AI agents and LLM applications with MLflow.

Updated Mar 10, 2026
Search Tags:#prompt optimization#llm#evaluation#agent testing#mlflow#genai#trace analysis
One-click install
npx skills add https://github.com/slysik/databricks-claude-coding --skill databricks-mlflow-evaluation-slysik
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/slysik/databricks-claude-coding/tree/main/.agents/skills/databricks-mlflow-evaluation
Command: npx skills add https://github.com/slysik/databricks-claude-coding --skill databricks-mlflow-evaluation-slysik

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill streamlines the evaluation of Generative AI agents and LLM applications, enabling users to write, debug, and optimize their GenAI code effectively using MLflow.

Core Features & Use Cases

  • Automated Evaluation: Run mlflow.genai.evaluate() for deterministic testing.
  • Scorer Integration: Use built-in scorers (Guidelines, Correctness, Safety, RetrievalGroundedness) or create custom ones.
  • Trace Analysis: Debug agent behavior by analyzing execution traces.
  • Prompt Optimization: Improve prompts using GEPA with optimize_prompts().
  • Use Case: You've built a RAG agent and want to ensure its responses are relevant, grounded in retrieved documents, and adhere to safety guidelines. This Skill provides the tools to set up an evaluation run, analyze the results, and iterate on prompt improvements.

Quick Start

Use the databricks-mlflow-evaluation skill to evaluate your agent using the provided evaluation dataset and safety and guidelines scorers.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate Generative AI agents using MLflow?▼

MLflow enables GenAI agent evaluation through the mlflow.genai.evaluate() function, which tests agents against datasets using built-in scorers for guidelines, correctness, safety, and retrieval groundedness.

Can I create custom scorers for LLM application testing in MLflow?▼

Yes, MLflow supports custom scorer integration alongside built-in scorers like Guidelines, Correctness, Safety, and RetrievalGroundedness, allowing tailored LLM application evaluation metrics.

What is the best way to debug GenAI agent behavior during evaluation?▼

Trace analysis via mlflow.search_traces() is the best way to debug GenAI agent behavior, allowing you to inspect execution traces and identify issues in agent logic.

How do I optimize prompts for LLM applications with MLflow?▼

You can optimize LLM application prompts using GEPA via mlflow.genai.optimize_prompts(), which iteratively improves prompt performance based on evaluation results.

Does MLflow support evaluation for RAG agents?▼

Yes, MLflow supports RAG agent evaluation by providing RetrievalGroundedness scorers to ensure responses are grounded in retrieved documents and adhere to safety guidelines.

What data schema is needed for running mlflow.genai.evaluate()?▼

Running mlflow.genai.evaluate() requires an evaluation dataset conforming to specific data schemas, with detailed guidance provided on schemas and scorer interfaces.

Related Skills

Explore More Software Engineering Skills→
itsadijmbtitsadijmbt
1

databricks-mlflow-evaluation

Run MLflow GenAI evaluations with dataset-driven scoring and built-in scorers.

Community
Advanced
alessandro9110alessandro9110
4

databricks-mlflow-evaluation

Evaluate, monitor, and optimize GenAI apps with MLflow.

Community
Advanced
manutejmanutej
61

mlops-workflows

Automate ML lifecycle management with MLflow experiment tracking and model registry.

Community
Advanced
bugrabilgebugrabilge
10

agent-evaluation

Design benchmark test suites for evaluating LLM agent reliability.

Community
Advanced