Evaluate and optimize LLM agents using MLflow's evaluation API.
npx skills add https://github.com/mlflow/skills --skill agent-evaluation-mlflow-2
Please help me install this Agent Skill. Skill: agent-evaluation Source: https://github.com/mlflow/skills/tree/main/agent-evaluation Command: npx skills add https://github.com/mlflow/skills --skill agent-evaluation-mlflow-2
This Skill provides a comprehensive workflow for evaluating and optimizing the performance of LLM agents, ensuring high-quality outputs and effective observability.
Run the following command to evaluate your agent: uv run python scripts/run_evaluation_template.py --module my_agent.agent --entry-point run_agent --dataset-name my-dataset
High-intent search queries and answers about installing and using this skill.
Evaluating LLM agents using MLflow involves a workflow of dataset creation, scorer definition, and result analysis. The process leverages MLflow's evaluation API to ensure traceability and observability of agent performance.
The best way to run a preliminary evaluation is to perform a dry run on a small subset of your dataset. This identifies and resolves potential issues using MLflow's evaluation API before executing the full evaluation.
Yes, MLflow supports registering custom scorers for LLM agent evaluation. You can utilize MLflow's scorer API to register and apply quality scorers to measure and optimize the performance of your agents.
To prepare datasets for evaluating your LLM agent, you can use provided tools to discover existing datasets or create new ones. These datasets are tailored to your agent's specific purpose to ensure accurate evaluation results.
Yes, you can analyze evaluation results to improve your LLM agent. The workflow analyzes the results from MLflow's evaluate API, identifies areas for improvement, and provides actionable recommendations for optimization.
ice-droid-99Instrument Python and TypeScript code with MLflow Tracing for AI workflows.
okwindsAudit repository compliance and generate markdown reports with Python scripts.
ethan-rioCreate, execute, and monitor LLM and code evaluators on Arize.
yo-stevenConvert JSON, CSV, or Hugging Face datasets into BYOB evaluation benchmarks with custom scorers.
