What problem does it solve? LLM applications in production lack visibility into why outputs are good or bad, how much they cost, and when quality degrades. This Skill provides instrumentation patterns and scripts to track token usage, latency, errors, and user feedback across LLM providers. ## Core Features & Use Cases - Cost and Token Tracking: Log every LLM call with per-model pricing, aggregate daily spend, and detect cost anomalies with a SQLite-backed monitor script. - Tracing Integration: Instrument pipelines with LangSmith, Langfuse v3, or OpenLLMetry to capture retrieval spans, generations, and agent tool calls. - Quality and Feedback Monitoring: Collect thumbs-up/down and 1-5 ratings, detect quality degradation against baselines, and compare prompt variants in A/B tests. - Use Case: A team running a RAG chatbot uses the monitor script to log calls, sets alerts for TTFT spikes and budget overruns, and reviews weekly quality reports to catch hallucination increases. ## Quick Start Use the llm-observability skill to set up cost tracking and latency monitoring for my OpenAI-based chat application.
