What problem does it solve? Automated evaluation of LLM outputs is unreliable when judges suffer from position bias, length bias, and uncalibrated scoring. This Skill provides production-grade patterns for building LLM-as-judge systems that correlate with human judgment. ## Core Features & Use Cases - Direct Scoring & Pairwise Comparison: Implements both evaluation approaches with structured prompts, chain-of-thought justification, and position-swap protocols to neutralize bias. - Rubric Generation: Creates calibrated scoring rubrics with level descriptions, edge cases, and domain-specific terminology, reducing evaluation variance by 40-60%. - Bias Mitigation & Metric Selection: Provides frameworks for detecting position, length, and self-enhancement bias, plus guidance on choosing metrics like Spearman's ρ, Cohen's κ, and agreement rates. - Use Case: When A/B testing two prompt variants, use pairwise comparison with position swapping to determine which produces better responses, with confidence scores calibrated to cross-pass consistency. ## Quick Start Ask the agent to compare two model responses for a given prompt using pairwise comparison with position bias mitigation and return a JSON verdict with confidence.
