What problem does it solve?
Recording feedback on LLM and agent spans often produces inconsistent, unfilterable labels that cannot be aggregated or audited later. This Skill enforces a disciplined annotation practice so every judgment is grounded, consistently named, and reusable across review sessions.
Core Features & Use Cases
- Config-driven annotation: Pulls the project's existing annotation configs via GraphQL and reuses their names, labels, and score bounds instead of inventing ad-hoc rubrics, and creates or extends configs when a new failure category emerges.
- Grounded judgment rules: Enforces one dimension per annotation, targeting the most specific responsible span, judging the first failure rather than downstream symptoms, and writing evidence-based explanations.
- Coaching mode: Teaches users to build a failure taxonomy from real traces through open coding and axial coding before codifying categories as annotation configs.
- Use Case: After debugging a failing agent run, ask the assistant to annotate the retrieval spans with a relevance label; it reuses the project's existing config, batches the annotations into one
ui.spans.annotate call, and reports the label distribution with links to annotated spans.
Quick Start
Ask the assistant to annotate the failing spans in this trace with the project's relevance rubric and explain each judgment.
