skills.rest
jpoindexter
jpoindexterCOMMUNITY
Explore all skills by this author →
Website𝕏@microninkGitHub

llm-observability-and-cost

Track LLM calls with traces, token counts, latency, and costs.

1|Updated Jun 9, 2026
Search Tags:#monitoring#drift detection#LLM#observability#cost tracking
One-click install
npx skills add https://github.com/jpoindexter/design-and-ai-skills --skill llm-observability-and-cost
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-observability-and-cost
Source: https://github.com/jpoindexter/design-and-ai-skills/tree/main/ai-engineering-skills/llm-observability-and-cost
Command: npx skills add https://github.com/jpoindexter/design-and-ai-skills --skill llm-observability-and-cost

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires opentelemetry, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill offers a comprehensive solution to observe and attribute costs to production Large Language Models (LLMs), ensuring efficient operation, error identification, and financial tracking.

Core Features & Use Cases

  • Observability: Tracks and records every LLM call with detailed traces, capturing model versions, prompts, responses, token counts, latency, and costs.
  • Cost Tracking: Attributes costs per call based on tokens, supporting granular billing and cost management.
  • Quality & Drift Detection: Monitors the quality of LLM responses and detects silent regressions to maintain service excellence.

Quick Start

Enable this Skill for detailed cost attribution and observability for your LLM systems, including real-time alerting and tracing.

Frequently Asked Questions about llm-observability-and-cost

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I track LLM token usage and attribute costs per API call in production?▼

LLM cost tracking attributes expenses per call based on token counts, capturing model versions, prompts, and responses to provide granular billing and comprehensive financial management for your production systems.

What is LLM drift detection and how does it prevent silent quality regressions?▼

LLM drift detection monitors response quality over time to identify silent regressions. By continuously assessing outputs, it maintains service excellence and alerts you before degradations impact user experience.

How do I implement observability for LLM systems using OpenTelemetry?▼

LLM observability with OpenTelemetry enables deep tracing of every model call, recording latency, token counts, and costs to deliver a comprehensive view of system performance and operational health.

Can I monitor LLM latency and model versions for every call in my application?▼

Yes, production monitoring records detailed traces for every LLM call, capturing model versions, prompt responses, and latency metrics to ensure efficient operation and rapid error identification across your application.

Does this LLM monitoring approach work for granular cost management and billing?▼

Yes, LLM cost tracking supports granular billing by attributing costs per call based on exact token usage, enabling precise cost management and financial tracking across all production model interactions.

What's the best way to set up real-time alerting and tracing for LLM production issues?▼

Production LLM observability provides real-time alerting and tracing by capturing detailed call traces, token counts, and latency metrics, enabling rapid error identification and efficient system operation.

Related Skills

Explore More Software Engineering Skills→
cedriczielcedricziel

opentelemetry-java

Instrument Java applications with OpenTelemetry for traces, metrics, and context propagation.

Community
Advanced
tense-itense-i
1

grafana-dashboards

Generate Grafana dashboards for visualizing Prometheus metrics.

Community
Intermediate
BagelHoleBagelHole
46

opentelemetry

Instrument applications and infrastructure with OpenTelemetry for unified traces, metrics, and logs.

Community
Intermediate
SelectXn00bSelectXn00b
198

model-usage

Track LLM token consumption and API costs from usage logs.

Official
Intermediate