analyzing-mlflow-session

You can debug MLflow multi-turn chat conversations by extracting per-turn inputs and outputs from root span attributes, linking traces by session ID to pinpoint exactly where conversations degrade. This allows you to review chat histories and spot conversational drift.

An MLflow session is a sequence of traces from a multi-turn chat conversation linked by a shared session ID in trace metadata. Trace analysis extracts inputs, outputs, and per-turn assessments from these traces to evaluate session quality and detect conversational drift.

You inspect individual MLflow chat session turns by running the discover_schema.sh script to identify session inputs and outputs, then using the inspect_turn.sh script to analyze specific turns and audit per-turn evaluations for issues.

Yes, you need jq and mlflow installed to analyze MLflow traces. These dependencies are required to fetch traces using the session ID and parse field paths like mlflow.traceInputs and mlflow.traceOutputs to reconstruct conversation turns.

Yes, you can find where a multi-turn chat went wrong by analyzing MLflow traces to audit session-level versus per-turn assessments. This process surfaces actionable insights and detects drift or inconsistencies across individual conversation turns.

A limitation when debugging MLflow session traces is that the analysis relies entirely on accessing existing traces and specific field paths like mlflow.spanInputs and mlflow.spanOutputs. If traces lack a shared session ID, conversation reconstruction will fail.