skills.rest
FMurray
FMurrayCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

agent-evaluation

Evaluates LLM agent output quality using MLflow datasets, scorers, and tracing.

Updated Mar 5, 2026
Search Tags:#llm evaluation#tracing#scorers#mlflow#agent evaluation#evaluation dataset#llm judge
One-click install
npx skills add https://github.com/FMurray/mlfts --skill agent-evaluation-fmurray
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-evaluation
Source: https://github.com/FMurray/mlfts/tree/main/.agents/skills/agent-evaluation
Command: npx skills add https://github.com/FMurray/mlfts --skill agent-evaluation-fmurray

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires mlflow, and includes scripts (resource) and references (resource) components.

What problem does it solve? LLM agents often produce wrong, incomplete, or inconsistent responses, and without a systematic evaluation workflow it is hard to measure quality, detect regressions, or justify prompt and tool changes. This Skill provides an end-to-end MLflow-based evaluation workflow so quality improvements are tracked, reproducible, and comparable over time. ## Core Features & Use Cases - Tracing Setup & Validation: Integrates MLflow autolog and @mlflow.trace decorators, then verifies traces with validation scripts before evaluation begins. - Scorer Registration: Discovers existing registered scorers, selects built-in scorers, and creates custom LLM judges via make_judge(), all registered in the MLflow experiment. - Dataset Management: Discovers existing evaluation datasets first to avoid duplication, then generates dataset creation scripts using mlflow.genai.datasets APIs. - Evaluation Execution & Analysis: Generates evaluation scripts with correctly-typed predict_fn wrappers, runs mlflow.genai.evaluate, and produces an analysis report with pass rates and improvement suggestions. - Use Case: You have a customer-support agent that sometimes gives incomplete answers. Use this Skill to trace the agent, register correctness and completeness scorers, build an evaluation dataset, and run mlflow.genai.evaluate to quantify quality before and after a prompt change. ## Quick Start Ask the AI to evaluate your agent's output quality with MLflow by setting up tracing, registering scorers, preparing a dataset, and running the evaluation workflow.

Frequently Asked Questions about agent-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate an LLM agent with MLflow?▼

Follow a four-step workflow: run the agent and inspect traces, select and register scorers, discover or create an evaluation dataset, then run mlflow.genai.evaluate with a predict_fn wrapper. Results are analyzed with the provided analysis script to produce a report.

How do I create custom scorers for MLflow agent evaluation?▼

Create custom scorers using the mlflow.genai.judges make_judge() API, then register them with scorer.register() so they appear in mlflow scorers list. Built-in scorers from mlflow.genai.scorers can also be selected and registered for common quality criteria.

What MLflow version is required for genai evaluation?▼

MLflow version 3.8.0 or higher is required. You also need MLFLOW_TRACKING_URI and MLFLOW_EXPERIMENT_ID environment variables configured, and tracing must be verified working before running any evaluation.

Why does mlflow.genai.evaluate fail with my predict_fn wrapper?▼

MLflow calls predict_fn by unpacking the dataset inputs dict as keyword arguments, so the wrapper signature must match the input keys exactly. A common mistake is defining def wrapper(inputs) instead of def wrapper(query) when inputs contain a query key.

Can I reuse existing evaluation datasets in MLflow?▼

Yes, and you should always check first. Run the list_datasets.py script to discover existing datasets in the experiment, compare their characteristics, and select one before creating a new dataset with mlflow.genai.datasets APIs.

Why should I avoid writing a custom evaluation framework?▼

Custom evaluation loops bypass MLflow's tracking of datasets, scorers, traces, and results, losing all experiment observability. Using mlflow.genai.evaluate with registered scorers keeps everything versioned and comparable within the experiment.

Related Skills

Explore More Software Engineering Skills→
BlackkadderBlackkadder

mlflow-onboarding

Detect MLflow project type and recommend quickstart tutorials and integration steps.

Community
Intermediate
context-labscontext-labs
1.1k

halo-loop

Diagnose HALO trace data to surface harness failure modes with evidence.

Official
Advanced
coval-aicoval-ai
2

review-llm-annotations-and-improve-prompt

Analyze human versus LLM judge annotation discrepancies to refine evaluation prompts.

Official
Advanced
ContextJet-aiContextJet-ai
29

add-llm-evals

Add offline and online evaluation scoring to LLM and agent applications.

Official
Advanced