skills.rest
JingyiWng
JingyiWngCOMMUNITY
Explore all skills by this author →
GitHub

agent-evaluation

Orchestrate end-to-end GenAI agent evaluation with MLflow datasets, scorers, and tracing.

Updated Mar 25, 2026
Search Tags:#evaluation#tracing#scorers#mlflow#datasets#genai
One-click install
npx skills add https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher --skill agent-evaluation-jingyiwng
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-evaluation
Source: https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher/tree/main/.claude/skills/agent-evaluation
Command: npx skills add https://github.com/JingyiWng/databricks_ai_dev_kit_price_watcher --skill agent-evaluation-jingyiwng

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Use this skill to evaluate and improve or optimize an existing LLM agent's output quality using MLflow evaluation with datasets, scorers, and tracing. IMPORTANT - Always load the instrumenting-with-mlflow-tracing skill before starting any work. Covers end-to-end evaluation workflow or individual components (tracing setup, dataset creation, scorer definition, evaluation execution).

Core Features & Use Cases

  • End-to-end evaluation workflow
  • Tracing setup and environment configuration
  • Dataset creation and scoring integration
  • Using MLflow's native APIs for datasets, scorers, and evaluation
  • Reusable evaluation components for ongoing agent improvement

Quick Start

Run a sample agent session and review the resulting MLflow trace to begin the evaluation workflow.

Frequently Asked Questions about agent-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM agent output quality using MLflow?▼

MLflow genai evaluation measures and improves GenAI agent output quality by combining tracing, datasets, and scorers. It orchestrates an end-to-end workflow across these components to evaluate and optimize existing LLM agents.

What is the best way to set up tracing for GenAI agent evaluation?▼

Always load the instrumenting-with-mlflow-tracing skill before starting evaluation work. This configures the tracing environment so you can run a sample agent session and review the resulting MLflow trace to begin evaluation.

How do I create datasets and define scorers for MLflow evaluation?▼

Creating datasets and defining scorers for MLflow evaluation involves applying the platform's native genai APIs. These components map directly to individual stages within the end-to-end evaluation workflow for measuring agent performance.

Do I need to load the instrumenting-with-mlflow-tracing skill before evaluating agents?▼

Yes, you must always load the instrumenting-with-mlflow-tracing skill before starting any agent evaluation work. This prerequisite ensures tracing is properly configured to capture the agent sessions needed for measuring output quality.

Can I evaluate individual components like scorers instead of running a full end-to-end agent evaluation?▼

Yes, MLflow evaluation applies to individual components like tracing setup, dataset creation, scorer definition, or evaluation execution independently. You can target specific stages rather than running the entire end-to-end evaluation workflow.

When should I not use MLflow genai APIs for agent evaluation?▼

MLflow genai APIs are designed specifically for evaluating GenAI agents using datasets, scorers, and tracing. If your workflow lacks these components or does not require measuring LLM agent output quality, alternative evaluation methods may be more suitable.

Related Skills

Explore More Data & Analytics Skills→
m19cm19c

agent-evaluation

Evaluate LLM agent output quality using MLflow native APIs.

Community
Advanced
mirakuimirakui

agent-evaluation

Evaluate LLM agent output quality using MLflow tracing, datasets, scorers, and evaluation APIs.

Community
Advanced
PaldomPaldom
1

agent-evaluation

Evaluate and improve LLM agent output quality using MLflow APIs.

Community
Advanced
BlackkadderBlackkadder

agent-evaluation

Evaluate and optimize LLM agent output quality using MLflow evaluation APIs.

Community
Advanced