skills.rest
mkgs-databricks-demos
mkgs-databricks-demosOFFICIAL
Explore all skills by this author →
GitHub

agent-evaluation

Evaluate GenAI agent outputs using MLflow APIs for datasets, scorers, and evaluation.

1|1|Updated Oct 1, 2025
Search Tags:#evaluation#tracing#scorers#mlflow#datasets#evaluate#genai
One-click install
npx skills add https://github.com/mkgs-databricks-demos/synthea-on-fhir --skill agent-evaluation-mkgs-databricks-demos
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-evaluation
Source: https://github.com/mkgs-databricks-demos/synthea-on-fhir/tree/main/.cursor/skills/agent-evaluation
Command: npx skills add https://github.com/mkgs-databricks-demos/synthea-on-fhir --skill agent-evaluation-mkgs-databricks-demos

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Evaluate GenAI agent outputs to identify quality gaps and optimization opportunities.

Core Features & Use Cases

  • End-to-end MLflow-based agent evaluation workflow covering tracing, datasets, scorers, and evaluation execution.
  • Guidance to register scorers, discover datasets, and analyze results for actionable improvements.
  • Templates and scripts to enforce MLflow API usage and ensure observability and reproducibility.

Quick Start

Run the provided MLflow-based agent evaluation templates to begin a complete evaluation.

Frequently Asked Questions about agent-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate GenAI agent outputs using MLflow?▼

To evaluate GenAI agent outputs using MLflow, you execute an end-to-end workflow covering tracing setup, dataset creation, scorer definition, and evaluation execution to identify quality gaps and optimization opportunities.

What is the best way to set up tracing for LLM agents in MLflow?▼

Setting up tracing for LLM agents in MLflow involves using MLflow's native APIs alongside provided templates to ensure observability and reproducibility across diverse agents and use cases.

Can I define custom scorers for my GenAI evaluation datasets in MLflow?▼

Yes, you can define custom scorers for GenAI evaluation datasets in MLflow by registering them through MLflow's native APIs to measure specific quality gaps in your agent outputs.

Does MLflow evaluation work for individual agent components or only end-to-end workflows?▼

MLflow evaluation works for both end-to-end evaluation workflows and individual components, allowing you to target tracing setup, dataset creation, scorer definition, and evaluation execution across diverse agents.

Why do I need to register scorers before running MLflow evaluate on my agents?▼

Registering scorers before running MLflow evaluate ensures your GenAI agents are scored consistently using MLflow's native APIs, which enforces observability, reproducibility, and accurate identification of optimization opportunities.

Related Skills

Explore More Data & Analytics Skills→
JingyiWngJingyiWng

agent-evaluation

Orchestrate end-to-end GenAI agent evaluation with MLflow datasets, scorers, and tracing.

Community
Advanced
leary-pokenleary-poken
1

databricks-mlflow-evaluation

Run MLflow GenAI evaluate() to score agent outputs with configurable scorers.

Community
Advanced
aaronachermannaaronachermann

agent-evaluation

Evaluate GenAI agents with MLflow datasets, scorers, and tracing.

Community
Advanced
BlackkadderBlackkadder

agent-evaluation

Evaluate and optimize LLM agent output quality using MLflow evaluation APIs.

Community
Advanced