What problem does it solve? AI features ship without proof they work: prompts regress silently, models hallucinate prices or facts, and teams pick models from leaderboards instead of measuring cost-latency-quality on their own inputs. This Skill turns 'I tried it a few times, seems fine' into a rigorous eval practice with hard safety gates and regression detection. ## Core Features & Use Cases - Pre-design model capability assessment: Define the model's job in one sentence, bucket outputs into must-never-happen gates vs. quality dimensions, and choose model tiers and providers per layer (STT, reasoning, TTS) by measuring on your own real inputs. - Golden set construction and grading ladder: Build 20-50 row test sets from real user utterances with checkable expected outcomes, graded on a three-level ladder from deterministic code checks to calibrated LLM judges to human review. - Failure-mode taxonomy and production metrics: Name and count every failure, map it to a fix class (structural, provider swap, taxonomy, prompt, model tier), and instrument acceptance rate, regeneration rate, edit distance, and latency P95 after launch. - Use Case: Before shipping a voice shopping assistant, run a 20-query golden set with a hard gate of zero hallucinated products or prices, a binary rubric for grounded recommendations judged by a different model family, and a 100% pass bar on medicine-deferral rows. ## Quick Start Ask the AI to write an eval spec for your chatbot feature, including a golden set of 20 real user inputs, hard safety gates, and pass bars for each test suite.
