skills.rest
robkisk
robkiskCOMMUNITY
Explore all skills by this author →
GitHub

databricks-mlflow-evaluation

Evaluate Generative AI agents with MLflow GenAI evaluation and trace analysis.

Updated Mar 10, 2026
Search Tags:#prompt optimization#llm#agent#evaluation#mlflow#genai#trace analysis
One-click install
npx skills add https://github.com/robkisk/coding-agents-databricks-app --skill databricks-mlflow-evaluation-robkisk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/robkisk/coding-agents-databricks-app/tree/main/.claude/skills/databricks-mlflow-evaluation
Command: npx skills add https://github.com/robkisk/coding-agents-databricks-app --skill databricks-mlflow-evaluation-robkisk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill streamlines the evaluation of Generative AI agents and LLM applications, enabling rigorous quality assessment, performance optimization, and alignment with domain expert preferences.

Core Features & Use Cases

  • Automated Evaluation: Run mlflow.genai.evaluate() with custom or built-in scorers.
  • Trace Analysis: Debug agent behavior using detailed MLflow traces.
  • Judge Alignment: Align LLM judges with domain expert feedback using MemAlign.
  • Prompt Optimization: Automatically improve prompts using GEPA.
  • Production Monitoring: Set up continuous evaluation of live traffic.
  • Use Case: Evaluate a RAG agent's groundedness and relevance, identify regressions after code changes, or optimize a prompt to reduce token usage while maintaining quality.

Quick Start

Use the databricks-mlflow-evaluation skill to evaluate your agent using the provided test dataset and safety and correctness scorers.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate generative AI agents using MLflow?▼

Run mlflow.genai.evaluate() with custom or built-in scorers to evaluate LLM application quality, using test datasets to measure groundedness and relevance.

How do I debug GenAI agent behavior using trace analysis?▼

Debug GenAI agent behavior by analyzing detailed MLflow traces stored in Unity Catalog, enabling you to identify regressions and inspect internal application logic.

Can I align LLM judges with domain expert feedback?▼

Yes, you can align LLM judges with domain expert feedback using MemAlign, ensuring automated evaluation results consistently match human expert preferences and quality standards.

What is the best way to optimize LLM prompts automatically?▼

Automatically optimize LLM prompts using GEPA to iteratively improve prompt quality, reduce token usage, and maintain application performance for generative AI outputs.

Does databricks-mlflow-evaluation support production monitoring?▼

Yes, the Skill supports production monitoring by enabling continuous evaluation of live traffic, allowing you to track GenAI agent performance and catch regressions.

Do I need Unity Catalog to store MLflow traces?▼

Unity Catalog integrates with the MLflow evaluation framework for trace storage and analysis, providing a centralized governance environment to debug agent behavior.

Related Skills

Explore More Software Engineering Skills→
databrickslabsdatabrickslabs
38

databricks-mlflow-evaluation

Evaluate GenAI agents with MLflow 3 GenAI evaluation APIs and built-in scorers.

Official
Advanced
thbehthbeh

databricks-mlflow-evaluation

Evaluate MLflow 3 GenAI agents with automated workflows and Unity Catalog integration.

Community
Advanced
ice-droid-99ice-droid-99

databricks-model-serving

Deploy Databricks MLflow models and GenAI agents to Model Serving endpoints.

Community
Advanced
leary-pokenleary-poken
1

databricks-mlflow-evaluation

Run MLflow GenAI evaluate() to score agent outputs with configurable scorers.

Community
Advanced