What problem does it solve?
Teams building LLM or agent applications often don't know where to start with evaluation — they may have traces, prompts, RAG pipelines, or nothing at all. This Skill acts as a triage router that diagnoses the user's situation and points them to the correct evaluation workflow instead of letting them guess.
Core Features & Use Cases
- Structured Diagnosis: Asks four diagnostic questions covering data form, label availability, stakes, and domain knowledge to classify the user's scenario.
- Triage Routing: Maps the diagnosed situation to one of eight sub-workflows (eval design, metric design, human alignment, eval report, RAG eval, prompt regression, red teaming, or bootstrap) via a triage table.
- Precondition Enforcement: Blocks premature routing — for example, refusing prompt-regression before paired baseline/candidate outputs exist, and refusing to call anything production-ready without ≥50 human labels and calibration.
- Use Case: A user says "I have production agent logs but no labels — how do I evaluate quality?" The Skill diagnoses data=traces, labels=none, and routes them to the eval-design workflow, then outlines the path to metric design and later human calibration.
Quick Start
Ask the assistant to help you figure out how to start evaluating your LLM application and answer its four diagnostic questions about your data, labels, stakes, and domain knowledge.
