skills.rest
andregit2026
andregit2026COMMUNITY
Explore all skills by this author →
GitHub

databricks-mlflow-evaluation

Orchestrate MLflow GenAI agent evaluations with scorers and trace datasets.

Updated Aug 23, 2026
Search Tags:#evaluation#alignment#scorers#mlflow#datasets#genai#trace-ingestion
One-click install
npx skills add https://github.com/andregit2026/Databricks_DQ_Business --skill databricks-mlflow-evaluation-andregit2026
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/andregit2026/Databricks_DQ_Business/tree/main/.claude/skills/databricks-general-skill-mlflow-evaluation
Command: npx skills add https://github.com/andregit2026/Databricks_DQ_Business --skill databricks-mlflow-evaluation-andregit2026

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill provides a structured workflow to evaluate GenAI agents using MLflow GenAI evaluation APIs, guiding users to implement scorers, build evaluation datasets from traces, and manage trace ingestion and monitoring.

Core Features & Use Cases

  • Scorer integration: Define and register custom scorers (Guidelines, Correctness, Safety, RetrievalGroundedness) for GenAI evaluations.
  • Evaluation dataset construction: Build ground-truth datasets from production traces and reference material.
  • Trace ingestion & monitoring: Integrate trace ingestion into experiments and enable production monitoring via MLflow.

Quick Start

Run mlflow.genai.evaluate() on a prepared evaluation dataset to score a GenAI agent.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate GenAI agents using MLflow?▼

To evaluate GenAI agents using MLflow, you run mlflow.genai.evaluate() on a prepared evaluation dataset to score agent responses against configured metrics like Correctness and Safety.

What is trace ingestion in MLflow GenAI evaluation workflows?▼

Trace ingestion in MLflow GenAI evaluation workflows involves integrating production traces into experiments to build ground-truth datasets and enable ongoing agent monitoring.

How do I configure custom scorers for MLflow GenAI evaluations?▼

You configure custom scorers for MLflow GenAI evaluations by defining and registering metrics such as Guidelines, RetrievalGroundedness, Correctness, and Safety to assess agent outputs.

Can I build evaluation datasets from production traces in MLflow?▼

Yes, you can build evaluation datasets from production traces in MLflow by structuring inputs as {'inputs': {'query': '...'}} and aligning them with reference materials as ground-truth expectations.

Does MLflow support alignment workflows for GenAI agent evaluation?▼

MLflow supports alignment workflows for GenAI agent evaluation by orchestrating scorers, evaluation datasets, and trace workflows to compare agent responses across multiple configured metrics.

Related Skills

Explore More Data & Analytics Skills→
ThomazRossitoThomazRossito
31

databricks-mlflow-evaluation

Orchestrate MLflow GenAI evaluation pipelines for scoring and monitoring agent behavior.

Community
Advanced
marktantongcomarktantongco

sentry-setup-ai-monitoring

Configure Sentry AI Agent Monitoring for AI/LLM calls and agent executions.

Community
Intermediate
thbehthbeh

databricks-mlflow-evaluation

Evaluate MLflow 3 GenAI agents with automated workflows and Unity Catalog integration.

Community
Advanced
databricks-solutionsdatabricks-solutions
1.8k

mlflow-evaluation

Automate GenAI evaluation of ML agents with MLflow workflows.

Official
Advanced