skills.rest
mirakui
mirakuiCOMMUNITY
Explore all skills by this author →
Website𝕏@mirakuiGitHub

databricks-mlflow-evaluation

Configure Databricks MLflow GenAI evaluation workflows with mlflow.genai.evaluate().

Updated Sep 9, 2017
Search Tags:#evaluation#mlflow#datasets#databricks#genai#gepa#memalign
One-click install
npx skills add https://github.com/mirakui/dotfiles --skill databricks-mlflow-evaluation-mirakui
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/mirakui/dotfiles/tree/main/claude/skills/databricks-mlflow-evaluation
Command: npx skills add https://github.com/mirakui/dotfiles --skill databricks-mlflow-evaluation-mirakui

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill provides a complete, end-to-end workflow for evaluating Databricks MLflow GenAI agents using MLflow 3 GenAI evaluation patterns. It covers building evaluation datasets, selecting and configuring scorers, aligning judges with MemAlign, and applying GEPA for automated prompt improvement, including optional production promotion.

Core Features & Use Cases

  • End-to-end evaluation using mlflow.genai.evaluate() with built-in scorers (Guidelines, Correctness, Safety, RetrievalGroundedness) and custom scorers.
  • Dataset and trace workflows: build eval datasets from traces, ingest production traces into Unity Catalog, and drive production monitoring.
  • Judge alignment and optimization: align judges with MemAlign using domain-expert feedback and drive GEPA prompt optimization for production-ready prompts.
  • Production monitoring and governance: register and monitor scorers, query traces, and progressively improve prompts with conditional promotion.

Quick Start

Start by configuring a baseline evaluation with mlflow.genai.evaluate() using a simple eval dataset, then add alignment and optimization steps to tighten evaluation signals.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate GenAI agents in Databricks using MLflow?▼

Evaluate GenAI agents in Databricks by configuring MLflow 3 workflows with mlflow.genai.evaluate(), selecting built-in scorers like RetrievalGroundedness and Safety, and building evaluation datasets from production traces.

How does MemAlign align LLM judges for GenAI evaluation?▼

MemAlign aligns LLM judges by incorporating domain-expert feedback to calibrate evaluation signals. This ensures scorers accurately reflect domain-specific quality requirements before applying automated prompt optimization.

Can I optimize evaluation prompts automatically with GEPA in Databricks?▼

Yes, GEPA drives automated prompt optimization by iteratively refining prompts based on evaluation results. Optimized prompts can then be conditionally promoted to production within your Databricks environment.

How do I build evaluation datasets from production traces in Unity Catalog?▼

Build evaluation datasets from production traces by ingesting trace data into Unity Catalog. This integration supports querying traces, registering scorers, and driving continuous production monitoring for repeatable evaluation cycles.

What scorers are available for MLflow GenAI evaluation?▼

MLflow GenAI evaluation provides built-in scorers including Guidelines, Correctness, Safety, and RetrievalGroundedness. You can also configure custom scorers to assess specific agent behaviors and domain requirements.

Do I need Databricks to run MLflow GenAI evaluation workflows?▼

The workflow targets Databricks MLflow 3 GenAI evaluation, leveraging Unity Catalog integration and trace ingestion for production monitoring. It is designed specifically for the Databricks environment.

Related Skills

Explore More Data & Analytics Skills→
databrickslabsdatabrickslabs
38

databricks-mlflow-evaluation

Evaluate GenAI agents with MLflow 3 GenAI evaluation APIs and built-in scorers.

Official
Advanced
Zack2626-okZack2626-ok

databricks-mlflow-evaluation

Automates MLflow GenAI evaluation with built-in or custom scorers and GEPA prompt optimization.

Community
Advanced
robkiskrobkisk

databricks-mlflow-evaluation

Evaluate Generative AI agents with MLflow GenAI evaluation and trace analysis.

Community
Advanced
aaronachermannaaronachermann

databricks-mlflow-evaluation

Evaluate GenAI agent outputs with MLflow GenAI evaluation APIs.

Community
Advanced