skills.rest
Aradhya0510
Aradhya0510COMMUNITY
Explore all skills by this author →
WebsiteGitHub

databricks-mlflow-evaluation

Evaluate Generative AI agents with MLflow 3 GenAI APIs and custom scorers.

3|1|Updated May 12, 2025
Search Tags:#prompt optimization#llm evaluation#evaluation#agent testing#mlflow#genai#trace analysis
One-click install
npx skills add https://github.com/Aradhya0510/databricks-cv-accelerator --skill databricks-mlflow-evaluation-aradhya0510
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/Aradhya0510/databricks-cv-accelerator/tree/main/.github/skills/databricks-mlflow-evaluation
Command: npx skills add https://github.com/Aradhya0510/databricks-cv-accelerator --skill databricks-mlflow-evaluation-aradhya0510

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill streamlines the evaluation of Generative AI agents and LLM applications, enabling rigorous quality assessment, debugging, and performance optimization.

Core Features & Use Cases

  • Automated Evaluation: Run mlflow.genai.evaluate() with built-in or custom scorers.
  • Trace Analysis: Debug agent behavior using detailed trace data.
  • Prompt Optimization: Automatically improve prompts using GEPA with aligned judges.
  • Production Monitoring: Continuously score live traffic with registered scorers.
  • Use Case: Evaluate a RAG agent's groundedness and relevance, then use the aligned judge and GEPA to optimize its prompt for better accuracy and reduced hallucinations.

Quick Start

Use the databricks-mlflow-evaluation skill to evaluate my agent using the safety and correctness scorers.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate my GenAI agent using MLflow?▼

To evaluate GenAI agents with MLflow, use the mlflow.genai.evaluate() API to run assessments using built-in scorers or your own custom evaluation metrics.

What is the best way to optimize LLM prompts to reduce hallucinations?▼

Prompt optimization for LLMs is handled by using GEPA with aligned judges to automatically refine prompts, which improves agent accuracy and reduces hallucinations.

Can I debug agent behavior using trace analysis in MLflow?▼

Yes, trace analysis in MLflow allows you to debug Generative AI agent behavior by examining detailed trace data captured during the evaluation process.

How do I set up continuous production monitoring for LLM applications?▼

Continuous production monitoring for LLM applications is established by continuously scoring live traffic with your registered custom and built-in scorers.

Does MLflow support custom scorer development and judge alignment?▼

Yes, MLflow supports custom scorer development for tailored evaluations and includes MemAlign to align judges, ensuring accurate scoring for your Generative AI agents.

How do I ingest evaluation traces into Unity Catalog?▼

Trace ingestion into Unity Catalog allows you to store and manage your detailed Generative AI agent evaluation traces directly within your data catalog.

Related Skills

Explore More Software Engineering Skills→
mirakuimirakui

databricks-mlflow-evaluation

Configure Databricks MLflow GenAI evaluation workflows with mlflow.genai.evaluate().

Community
Advanced
prashsubprashsub
1

production-monitoring

Monitor GenAI agent outputs using MLflow registered scorers and archived traces.

Community
Advanced
prashsubprashsub
1

mlflow-genai-evaluation

Automate Generative AI agent evaluation with MLflow GenAI and custom scorers.

Community
Advanced
ScottHMcKeanScottHMcKean

databricks-mlflow-evaluation

Evaluate and monitor Generative AI applications with MLflow.

Community
Advanced