What problem does it solve? Evaluating LLM outputs at scale is unreliable when judges suffer from position bias, length bias, and inconsistent scoring. This Skill provides production-grade patterns for building LLM-as-judge systems that produce consistent, calibrated, and auditable quality assessments. ## Core Features & Use Cases - Direct Scoring & Pairwise Comparison: Implement evidence-first scoring prompts and position-swapped pairwise comparisons with consistency checks and confidence calibration. - Bias Mitigation: Apply concrete techniques against position, length, self-enhancement, verbosity, and authority biases, including cross-model evaluation and length-normalized scoring. - Rubric Generation: Generate domain-specific scoring rubrics with level descriptions, edge cases, and strictness calibration to reduce evaluation variance. - Use Case: When A/B testing a new prompt, run pairwise comparisons with position swapping to pick the winning variant, then validate judge reliability against human labels using Spearman's rho and Cohen's kappa. ## Quick Start Ask the AI to compare two model responses for a given prompt using pairwise comparison with position-swap bias mitigation and return a winner with calibrated confidence.
