skills.rest
ScottHMcKean
ScottHMcKeanCOMMUNITY
Explore all skills by this author →
GitHub

databricks-mlflow-evaluation

Evaluate and monitor Generative AI applications with MLflow.

3|Updated Jan 2, 2025
Search Tags:#monitoring#prompt optimization#evaluation#mlflow#genai#trace analysis
One-click install
npx skills add https://github.com/ScottHMcKean/genai_notebooks --skill databricks-mlflow-evaluation-scotthmckean
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-mlflow-evaluation
Source: https://github.com/ScottHMcKean/genai_notebooks/tree/main/.cursor/skills/databricks-mlflow-evaluation
Command: npx skills add https://github.com/ScottHMcKean/genai_notebooks --skill databricks-mlflow-evaluation-scotthmckean

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill streamlines the complex process of evaluating and monitoring Generative AI applications using MLflow, ensuring your models meet quality, safety, and performance standards.

Core Features & Use Cases

  • End-to-End Evaluation: Run mlflow.genai.evaluate() for comprehensive quality checks.
  • Custom Scorers: Build project-specific metrics using @scorer or Scorer classes.
  • Trace Analysis: Debug agent behavior and identify regressions using mlflow.search_traces().
  • Production Monitoring: Set up continuous evaluation of live traffic with registered scorers.
  • Judge Alignment: Align LLM judges with domain expert preferences using MemAlign.
  • Prompt Optimization: Automatically improve prompts with GEPA using optimize_prompts().
  • Use Case: You've deployed a new RAG agent. Use this Skill to evaluate its retrieval groundedness, correctness against expected facts, and adherence to safety guidelines, then set up continuous monitoring to track performance in production.

Quick Start

Use the databricks-mlflow-evaluation skill to run an evaluation using the provided dataset and predict function.

Frequently Asked Questions about databricks-mlflow-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate Generative AI applications using MLflow?▼

You can build project-specific custom scorers in MLflow by using the `@scorer` decorator or defining custom `Scorer` classes. This allows you to tailor evaluation metrics precisely to your application's unique quality, safety, and performance requirements.

How do I debug GenAI agent behavior using MLflow trace analysis?▼

Debug GenAI agent behavior and identify regressions by querying execution data with `mlflow.search_traces()`. This trace analysis functionality allows you to inspect step-by-step agent interactions and pinpoint specific failure points within your models.

Can I automatically optimize prompts using MLflow?▼

Yes, you can automatically optimize prompts using MLflow by leveraging GEPA through the `optimize_prompts()` function. This automates the prompt improvement process, iteratively refining inputs to enhance overall model performance.

How do I align LLM judges with domain expert preferences in MLflow?▼

Align LLM judges with domain expert preferences in MLflow by utilizing the MemAlign technique. This ensures your automated evaluation judges accurately reflect human expert standards when scoring model outputs during quality checks.

Does MLflow support continuous monitoring for live GenAI traffic?▼

Yes, MLflow supports continuous production monitoring for live GenAI traffic by setting up registered scorers. This enables ongoing evaluation of live traffic to track performance, detect regressions, and ensure safety standards are maintained.

Related Skills

Explore More Software Engineering Skills→
databricks-solutionsdatabricks-solutions
1.8k

mlflow-evaluation

Automate GenAI evaluation of ML agents with MLflow workflows.

Official
Advanced
chiruu12chiruu12
1

canary

Monitor production apps for errors and regressions after deployment.

Community
Advanced
qcmuuqcmuu

mlflow

Track ML experiments, version models, and deploy via MLflow registry URIs.

Community
Advanced
prashsubprashsub
1

mlflow-genai-evaluation

Automate Generative AI agent evaluation with MLflow GenAI and custom scorers.

Community
Advanced