analyzing-mlflow-session

Analyzes multi-turn MLflow chat sessions by reconstructing traces and assessments.

Updated Mar 5, 2026
One-click install
npx skills add https://github.com/FMurray/mlfts --skill analyzing-mlflow-session-fmurray
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: analyzing-mlflow-session
Source: https://github.com/FMurray/mlfts/tree/main/.agents/skills/analyze-mlflow-chat-session
Command: npx skills add https://github.com/FMurray/mlfts --skill analyzing-mlflow-session-fmurray

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? Debugging multi-turn chat conversations is hard when failures originate in earlier turns and propagate. This Skill reconstructs an entire MLflow session from its traces, pinpoints the turn where things went wrong, and surfaces assessment signals without reading every trace in full. ## Core Features & Use Cases - Session Reconstruction: Discovers the input/output schema from the first trace, then extracts inputs, outputs, and assessments across all session traces using the MLflow CLI. - Assessment Interpretation: Distinguishes session-level from per-turn assessments, filters out scorer errors, and reads rationale fields to interpret values correctly. - Root-Cause Analysis: Correlates trace behavior with codebase patterns like context window management and cross-turn state. - Use Case: A user reports a wrong answer on turn 5 of a chatbot conversation. The Skill reconstructs the session, finds the factual error actually originated in turn 3, and traces it to a retriever returning an outdated document. ## Quick Start Ask the assistant to analyze the MLflow session with a given session ID and experiment ID to find where the conversation went wrong.

Frequently Asked Questions about analyzing-mlflow-session

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I analyze an MLflow chat session across multiple turns?

Search traces filtered by the mlflow.trace.session metadata key, ordered by timestamp. First fetch one full trace to discover the input/output schema, then use --extract-fields on the search command to pull those fields across all session traces efficiently.

How do I filter MLflow traces by session ID?

Use mlflow traces search with --filter-string 'metadata.`mlflow.trace.session` = "<SESSION_ID>"'. The metadata key contains dots, so it must be escaped with backticks, and the outer string should use single quotes to avoid shell command substitution.

Why does mlflow traces search produce no output when piped?

Large CLI output exceeds the Bash tool's pipe limit and gets silently redirected. Always redirect MLflow CLI output to a file first with > /tmp/output.json, then process the file with jq or other tools.

What is the difference between session-level and per-turn assessments in MLflow?

Session-level assessments are attached to the first trace and include mlflow.trace.session in their metadata, evaluating the conversation as a whole. Per-turn assessments lack that metadata key and evaluate individual turns only.

Does an assessment error mean the MLflow trace failed?

No. A feedback.error field on an assessment means the scorer or judge failed, not the trace itself. Filter out assessments with errors before using them to identify problematic turns.