aio11y

What problem does it solve? Debugging LLM-powered applications in production requires visibility into what agents actually did and whether their output was any good. This Skill provides command-line access to Grafana's AI Observability platform so you can search conversations, inspect individual generations, and set up automated quality scoring without leaving your terminal. ## Core Features & Use Cases - Conversation & Generation Inspection: List, search, and filter conversations by agent, model, status, error type, and more to debug what an agent did in production. - Evaluator Management: Create, test, and delete evaluators (LLM judge, regex, heuristic, JSON schema) using YAML definitions, with built-in templates as starting points. - Evaluation Rules & Scores: Bind evaluators to production traffic with selectors, match filters, and sampling rates, then check resulting scores per generation. - Use Case: An agent starts returning malformed JSON in production. Search conversations with status = "error" filters, inspect the failing generations, then create a json_schema evaluator and a rule to continuously score output format compliance. ## Quick Start Ask the assistant to list recent AI Observability conversations for a specific agent using the gcx aio11y conversations search command.