skills.rest
datasciencemonkey
datasciencemonkeyCOMMUNITY
Explore all skills by this author →
GitHub

databricks-mlflow-evaluation

Automate MLflow GenAI evaluation workflows for Databricks agents with structured datasets and reference files.

27|9|Updated Feb 3, 2026
Search Tags:#evaluation#alignment#mlflow#datasets#production-monitoring#trace-analysis#genai
One-click install
npx skills add https://github.com/datasciencemonkey/coding-agents-databricks-apps --skill databricks-mlflow-evaluation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/datasciencemonkey/coding-agents-databricks-apps/tree/main/.claude/skills/databricks-mlflow-evaluation
Command: npx skills add https://github.com/datasciencemonkey/coding-agents-databricks-apps --skill databricks-mlflow-evaluation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

MLflow GenAI evaluation workflows in Databricks environments are often ad-hoc and hard to reproduce; this Skill provides a structured, reference-backed approach to validate prompts, scorers, and domain-aligned judges across the evaluation lifecycle.

Core Features & Use Cases

  • End-to-end evaluation patterns covering dataset creation, scorer configuration, trace analysis, and production monitoring for MLflow GenAI workloads.
  • Support for the full domain-expert optimization loop: evaluation, labeling, judge alignment (MemAlign), automated prompt optimization (GEPA), and conditional production promotion.
  • Built-in references and reference files (GOTCHAS, CRITICAL-interfaces, and patterns) to guide implementation, auditing, and repeatability.

Quick Start

Create a small eval dataset and run mlflow.genai.evaluate with your agent to observe baseline metrics and judge alignment.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up MLflow GenAI evaluation for Databricks agents?▼

Structured evaluation datasets are required because this toolkit enforces reference-backed patterns for validating prompts, scorers, and domain-aligned judges across the full MLflow GenAI evaluation lifecycle, ensuring reproducibility and auditing readiness.

How does judge alignment work in MLflow GenAI evaluation?▼

The domain-expert optimization loop covers evaluation, labeling, judge alignment via MemAlign, automated prompt optimization using GEPA, and conditional production promotion for Databricks agents. It enforces structured datasets and reference patterns throughout.

Can I use MLflow GenAI evaluation for production monitoring and trace analysis?▼

Built-in reference files include GOTCHAS, CRITICAL-interfaces, and patterns that guide implementation, auditing, and repeatability for MLflow GenAI evaluation workflows, replacing ad-hoc approaches with structured, reference-backed validation.

What is the best way to automate prompt optimization in MLflow GenAI evaluation?▼

This toolkit replaces ad-hoc MLflow GenAI evaluation workflows with a structured, reference-backed approach. It ensures repeatability by enforcing structured datasets, judge alignment, GEPA prompt optimization, and production monitoring with explicit reference patterns.

Related Skills

Explore More Data & Analytics Skills→
databricks-solutionsdatabricks-solutions
1.8k

mlflow-evaluation

Automate GenAI evaluation of ML agents with MLflow workflows.

Official
Advanced
mirakuimirakui

databricks-mlflow-evaluation

Configure Databricks MLflow GenAI evaluation workflows with mlflow.genai.evaluate().

Community
Advanced
Zack2626-okZack2626-ok

databricks-mlflow-evaluation

Automates MLflow GenAI evaluation with built-in or custom scorers and GEPA prompt optimization.

Community
Advanced
leary-pokenleary-poken
1

databricks-mlflow-evaluation

Run MLflow GenAI evaluate() to score agent outputs with configurable scorers.

Community
Advanced