What problem does it solve?
AI applications fail silently in production: the model returns HTTP 200 with a response that is wrong, off-topic, ignores instructions, or misuses tools, and these failures never raise an error. This Skill finds both loud and silent failures in PostHog LLM trace data and groups them into a ranked, actionable failure taxonomy.
Core Features & Use Cases
- Failure Trace Discovery: Select candidate traces using code errors, metric outliers (tokens, latency, cost), trace-type slices, stratified sampling, existing-eval spikes, or clustering.
- Trace Reading and Coding: Read 20-30 traces per use case with query-llm-trace, note what went wrong, and group notes into named failure modes.
- Ranked Failure Taxonomy: Deliver a ranked list of failure modes with deep-linked example traces, ready to turn into fixes or online evaluations.
- Use Case: A team notices their support chatbot gives bad answers. Use this Skill to scope to the support trace slice, read a batch of traces, and come back with ranked modes like "ignores the date filter" or "invents a policy", each with linked example traces.
Quick Start
Ask the assistant to find and categorize the failure modes in your AI feature's production traces from the last 7 days.
