What problem does it solve? LLM applications often ship on vibes — they seem to work, but nobody can say whether retrieval or generation is failing, whether a prompt change regressed quality, or whether spend is under control. This Skill takes an LLM app from unmeasured to gated: tracing every query, scoring quality with RAG metrics and LLM judges, gating deploys on eval thresholds, and enforcing production controls like prompt versioning, budgets, and input guardrails. ## Core Features & Use Cases - Tracing architecture: Dual instrumentation (callback handler plus decorator) producing trace-per-query spans grouped by session, with tokens, cost, and latency visible per named span. - Metric suite with retriever/generation split: Ragas-style metrics (context precision/recall, entities recall, noise sensitivity, faithfulness, response relevancy, tool call accuracy) that isolate whether failures come from retrieval or generation. - LLM-as-judge pipelines and CI gating: Judge evaluators wired to trace fields, hand-verified against labeled samples, then run in CI to block deploys when metrics drop below thresholds. - Production hardening: Prompt registry with version-linked traces, TTL-bounded chat history with token trimming, NeMo-style input rails, and gateway controls (virtual keys, budgets with durations, rpm/tpm limits) via LiteLLM. - Use Case: Your RAG chatbot is hallucinating in production. Use this Skill to add tracing, build a 50–200 example eval set from real traces, measure faithfulness versus context recall to pinpoint whether retrieval or generation is at fault, and gate future prompt changes on the metric suite. ## Quick Start Ask the agent to add tracing and a faithfulness-plus-context-recall eval suite to your RAG chatbot using Langfuse and Ragas, then gate deploys on the results.
