databricks-mlflow-evaluation

Use mlflow.genai.evaluate() with a dataset containing an inputs key per record, a predict_fn that receives unpacked keyword arguments, and a list of scorers such as Guidelines, Correctness, or Safety. Do not use the older mlflow.evaluate() API.

Decorate a function with @scorer from mlflow.genai.scorers and return a boolean, float, string, or Feedback object. The function can accept inputs, outputs, expectations, and the full trace object for deeper analysis.

Correctness requires ground truth in each record's expectations field, either expected_facts or expected_response. Without expectations, the scorer has nothing to compare the agent output against and will fail.

No. RetrievalGroundedness requires a span marked with span_type="RETRIEVER" in the trace. Decorate your retrieval function with @mlflow.trace(span_type="RETRIEVER") so the scorer can find the retrieved documents.

Lower scores after alignment are expected and indicate the judge now applies domain-expert standards instead of generic criteria. The unaligned judge was likely giving inflated scores, so the lower aligned score is a more accurate signal.

Unity Catalog trace ingestion requires mlflow[databricks]>=3.9.0, while optimize_prompts() requires MLflow 3.5.0 or later. You must also set MLFLOW_TRACING_SQL_WAREHOUSE_ID before linking a UC schema to an experiment.