llm-eval-designer

Evaluation design for LLM workflows involves systematically identifying failure modes like hallucination and overfitting, then building test cases and scorers to detect them. This Skill guides you through failure-mode analysis, generalization patterns, and scorer design to produce golden datasets and test-case templates that meet assay-kit compliance standards.

Common LLM failure modes include hallucination, overfitting, and partial processing. This Skill helps you enumerate these modes for your specific workflow, then design targeted test cases to expose each one. Comprehensive failure-mode coverage ensures robust evaluation of your LLM-based system.

Golden-dataset construction begins with test-case template generation based on your LLM task and identified failure modes. This Skill provides guidance on extending and diversifying test cases to support generalization testing, then structures outputs as golden-dataset.yaml for downstream validation and scoring.

Scorer design requires aligning your scoring logic with the failure modes and test cases you've defined. This Skill walks you through creating scorer configurations that work with your test-case templates, ensuring consistent and measurable evaluation of LLM outputs against your functional and technical requirements.

Yes. This Skill includes generalization pattern guidance to help you design test cases that expose how your LLM behaves under different conditions. Diversified test cases and comprehensive failure-mode coverage ensure your evaluation generalizes across model variations and edge cases.