skills.rest
ScottHMcKean
ScottHMcKeanCOMMUNITY
Explore all skills by this author →
GitHub

querying-mlflow-metrics

Fetch aggregated trace metrics from MLflow tracking servers.

3|Updated Jan 2, 2025
Search Tags:#llm#metrics#evaluation#latency#token usage#mlflow
One-click install
npx skills add https://github.com/ScottHMcKean/genai_notebooks --skill querying-mlflow-metrics-scotthmckean
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: querying-mlflow-metrics
Source: https://github.com/ScottHMcKean/genai_notebooks/tree/main/.cursor/skills/querying-mlflow-metrics
Command: npx skills add https://github.com/ScottHMcKean/genai_notebooks --skill querying-mlflow-metrics-scotthmckean

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill simplifies the process of retrieving and analyzing performance metrics from MLflow tracking servers, providing insights into token usage, latency, and quality evaluations.

Core Features & Use Cases

  • Fetch Aggregated Metrics: Retrieve metrics like token usage, latency, and trace counts.
  • Analyze Trends: Visualize token usage over time and latency percentiles.
  • Evaluate Quality: Assess LLM performance using quality scores and assessment counts.
  • Use Case: A data scientist needs to understand the token consumption of their LLM experiments over the last week. They can use this Skill to get a daily summary of total token usage.

Quick Start

Fetch the total token usage aggregated by SUM and AVG for experiment ID 1 from the MLflow server at http://localhost:5000.

Frequently Asked Questions about querying-mlflow-metrics

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I query MLflow trace metrics for LLM experiments?▼

To query MLflow trace metrics, you need to provide your MLflow server URL and experiment IDs to fetch aggregated data including token usage, latency, trace counts, and quality evaluations.

What is the best way to analyze LLM token usage trends in MLflow?▼

Analyzing LLM token usage trends in MLflow involves fetching aggregated trace metrics by SUM and AVG, allowing you to visualize daily total token consumption and latency percentiles over a specific timeframe.

Can I check LLM quality evaluations and latency percentiles from an MLflow tracking server?▼

Yes, you can check LLM quality evaluations and latency percentiles by querying your MLflow tracking server to retrieve assessment counts and quality scores alongside trace statistics.

Do I need specific experiment IDs to retrieve trace statistics from MLflow?▼

Yes, you need specific experiment IDs and the MLflow server URL to retrieve aggregated trace statistics, as these parameters are required to target and query the correct LLM evaluation data.

Why does querying MLflow metrics require a server URL?▼

Querying MLflow metrics requires a server URL because the Skill connects directly to your MLflow tracking server to fetch aggregated trace data, token usage, and latency statistics for the specified experiments.

Related Skills

Explore More Data & Analytics Skills→
BlackkadderBlackkadder

querying-mlflow-metrics

Fetch aggregated trace metrics from MLflow tracking servers.

Community
Intermediate
PaldomPaldom
1

querying-mlflow-metrics

Fetch aggregated trace metrics from MLflow tracking servers.

Community
Intermediate
slysikslysik

querying-mlflow-metrics

Fetch aggregated trace metrics from MLflow tracking servers.

Community
Intermediate
LaurentPRAT-DBLaurentPRAT-DB

querying-mlflow-metrics

Aggregates and charts MLflow usage metrics for specified experiments.

Community
Intermediate