skills.rest
choijinwon
choijinwonCOMMUNITY
Explore all skills by this author →
GitHub

agent-evaluation

Evaluate and optimize LLM agent performance using MLflow.

1|Updated Jun 18, 2026
Search Tags:#llm#performance optimization#mlflow#evaluation workflow#agent evaluation
One-click install
npx skills add https://github.com/choijinwon/opecode-ml-skill- --skill agent-evaluation-choijinwon
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-evaluation
Source: https://github.com/choijinwon/opecode-ml-skill-/tree/main/.agents/skills/agent-evaluation
Command: npx skills add https://github.com/choijinwon/opecode-ml-skill- --skill agent-evaluation-choijinwon

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires mlflow, mlflow.genai, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a comprehensive guide to evaluate and optimize the performance of LLM agents using MLflow, addressing various aspects such as tool selection accuracy, answer quality, and cost reduction.

Core Features & Use Cases

  • Systematic Evaluation: Evaluates agents using MLflow with datasets, scorers, and tracing, covering end-to-end workflows or individual components.
  • Scorer Selection: Offers built-in scorers for relevance, completeness, and accuracy, with options for custom scorers.
  • Dataset Preparation: Assists in discovering and preparing datasets for evaluation.
  • Evaluation Execution: Automates the evaluation process, applying scorers to traces and analyzing results.
  • Analysis Report: Generates a markdown evaluation report with pass rates, failure patterns, and recommendations.
  • Use Case: Imagine you have an LLM agent that you want to improve. Use this Skill to evaluate its performance using various datasets and scorers, and get detailed insights into its strengths and weaknesses.

Quick Start

Run the 'agent-evaluation' skill to start the evaluation process for your LLM agent.

Frequently Asked Questions about agent-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM agent performance using MLflow?▼

You can evaluate LLM agent performance using MLflow by preparing datasets, defining scorers, and collecting traces. This skill automates the evaluation execution and generates a markdown report with pass rates, failure patterns, and recommendations.

What built-in metrics are available for agent evaluation in MLflow?▼

Agent evaluation in MLflow offers built-in scorers for relevance, completeness, and accuracy. You can also define custom scorers to measure specific aspects like tool selection accuracy and answer quality.

How do I set up an evaluation workflow for my LLM agent?▼

To set up an LLM agent evaluation workflow, you need MLflow and associated Python libraries for trace collection. You then prepare datasets, select appropriate scorers, and run the evaluation to analyze agent strengths and weaknesses.

Can I use custom scorers for evaluating LLM agents with MLflow?▼

Yes, you can use custom scorers for evaluating LLM agents with MLflow. While built-in scorers cover relevance, completeness, and accuracy, the evaluation workflow supports custom scorer definitions to apply specialized metrics to your traces.

What is the best way to optimize LLM agents and reduce costs?▼

The best way to optimize LLM agents and reduce costs is through systematic evaluation using MLflow. By analyzing evaluation reports on tool selection accuracy and answer quality, you can identify failure patterns and improve efficiency.

Related Skills

Explore More Software Engineering Skills→
itsadijmbtitsadijmbt
1

skill-test

Evaluate and optimize Databricks agent skills against ground truth YAML test cases.

Community
Advanced
mchien15mchien15

context-engineering

Analyze context utilization and risk factors to generate health scores and optimization recommendations.

Community
Advanced
ice-droid-99ice-droid-99

instrumenting-with-mlflow-tracing

Instrument Python and TypeScript code with MLflow Tracing for AI workflows.

Community
Intermediate
mlflowmlflow
69

agent-evaluation

Evaluate and optimize LLM agents using MLflow's evaluation API.

Official
Advanced