databricks-mlflow-evaluation

Automate MLflow GenAI evaluation and prompt optimization by using GEPA for automated prompt improvement alongside built-in or custom scorers. This Skill orchestrates the entire workflow to reduce manual model evaluation effort.

GEPA is an automated prompt improvement mechanism integrated into MLflow GenAI workflows. It systematically refines and optimizes prompts during the evaluation process to enhance model outputs without manual tuning.

Yes, storing and analyzing production traces for continuous monitoring requires Databricks and Unity Catalog. You also need MLflow version 3.1.0 or higher and the OpenAI library installed to support the trace ingestion workflow.

Yes, you can align GenAI judges with domain expert feedback using the MemAlign mechanism. This improves evaluation accuracy by calibrating automated judges to closely reflect human expert assessments during model evaluation workflows.

The best way to evaluate MLflow GenAI models with custom scorers is to run an automated evaluation workflow that integrates custom scoring logic with continuous production trace monitoring. This ensures accurate, context-specific model assessment.