production-monitoring

Perform on-demand assessment of GenAI agent performance by using the mlflow.genai.assess() function. This allows you to perform periodic or ad-hoc evaluations of agent response quality, safety, and relevance outside of continuous monitoring.

Archive GenAI traces for historical debugging and analysis by automatically storing detailed trace information using Unity Catalog. This archival process preserves execution history, allowing you to investigate past agent issues and analyze metric trends over time.

You can retroactively calculate evaluation metrics for historical GenAI data using the metric backfill feature. This allows you to apply newly registered MLflow scorers to past traces, ensuring historical performance data aligns with current assessment criteria.

You need MLflow for scorer registration and assessment, and Unity Catalog for trace archival to fully implement this GenAI production monitoring strategy. These dependencies provide the necessary infrastructure for continuous evaluation and historical trace storage.

Manage the lifecycle of MLflow scorers for GenAI monitoring by utilizing the built-in scorer lifecycle management features. This handles the registration, updating, and retirement of scorers, ensuring your production evaluation criteria remain accurate and relevant over time.