skills.rest
aaronachermann
aaronachermannCOMMUNITY
Explore all skills by this author →
GitHub

databricks-mlflow-evaluation

Evaluate GenAI agent outputs with MLflow GenAI evaluation APIs.

Updated Apr 18, 2026
Search Tags:#evaluation#traces#scorers#mlflow#genai#gepa#memalign
One-click install
npx skills add https://github.com/aaronachermann/PolentaEncoders --skill databricks-mlflow-evaluation-aaronachermann
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/aaronachermann/PolentaEncoders/tree/main/.github/skills/databricks-mlflow-evaluation
Command: npx skills add https://github.com/aaronachermann/PolentaEncoders --skill databricks-mlflow-evaluation-aaronachermann

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Establish reproducible, production-ready evaluation for GenAI agents by providing patterns, critical API facts, and end-to-end workflows for running mlflow.genai.evaluate(), building datasets from traces, aligning LLM judges with SME feedback, and automating prompt optimization.

Core Features & Use Cases

  • Evaluation Workflows: Step-by-step patterns for first-time setup, production trace ingestion, regression detection, and performance debugging.
  • Scorers & Judges: Guidance for built-in scorers (Guidelines, Correctness, Safety, RetrievalGroundedness), custom scorer patterns, and MemAlign judge alignment.
  • Optimization & Monitoring: Instructions for GEPA prompt optimization, creating MLflow-managed datasets, and enabling Unity Catalog trace ingestion and production monitoring.

Quick Start

Run a quick evaluation by calling mlflow.genai.evaluate() with a local predict_fn that accepts unpacked inputs and the desired scorers to generate traces and aggregate metrics.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate GenAI agent outputs using MLflow?▼

Evaluate GenAI agent outputs by calling mlflow.genai.evaluate() with a local predict_fn that accepts unpacked inputs and applying built-in or custom scorers to generate traces and aggregate metrics.

How do I align LLM judges with SME feedback for GenAI evaluation?▼

Align LLM judges with SME feedback using MemAlign, which requires the label schema name to match the judge name exactly to ensure proper alignment during evaluation.

Can I build an evaluation dataset from production traces in MLflow?▼

Build evaluation datasets from production traces by enabling Unity Catalog trace ingestion, satisfying nested inputs schema requirements and mlflow[databricks] version constraints for trace ingestion.

What is the best way to automate prompt optimization for GenAI scorers?▼

Automate prompt optimization for GenAI scorers by running GEPA, which iteratively refines prompts to improve evaluation metrics while adhering to correct scorer and judge interfaces.

Does the MLflow GenAI evaluation API support custom scorers?▼

The MLflow GenAI evaluation API supports custom scorers alongside built-in options like Guidelines, Correctness, Safety, and RetrievalGroundedness, provided they implement the correct scorer interface.

Why does my predict_fn fail during MLflow GenAI evaluation?▼

Your predict_fn likely fails during MLflow GenAI evaluation if it does not accept unpacked inputs as kwargs, which is a strict technical requirement for the API to process nested inputs correctly.

Related Skills

Explore More Data & Analytics Skills→
BlackkadderBlackkadder

databricks-mlflow-evaluation

Run MLflow GenAI evaluations with custom scorers, judges, and trace datasets.

Community
Advanced
databricks-solutionsdatabricks-solutions
1.8k

mlflow-evaluation

Automate GenAI evaluation of ML agents with MLflow workflows.

Official
Advanced
RamVegirajuRamVegiraju
4

databricks-mlflow-evaluation

Evaluate GenAI agents with MLflow GenAI APIs for safety, correctness, relevance, and groundedness.

Community
Advanced
leary-pokenleary-poken
1

databricks-mlflow-evaluation

Run MLflow GenAI evaluate() to score agent outputs with configurable scorers.

Community
Advanced