advanced-evaluation

Choose direct scoring when objective ground truth exists (factual accuracy, instruction following) and pairwise comparison for subjective preferences (tone, style). Always require evidence-based justification before scores, which improves reliability by 15-25%.

Run the comparison twice with swapped response positions, then check consistency. If both passes agree on the winner, average the confidences; if they disagree, return TIE with 0.5 confidence since position bias corrupted the result.

Use direct scoring for objective criteria like factual accuracy and format compliance. Use pairwise comparison for subjective judgments like tone and persuasiveness, as it achieves higher human-judge agreement for preference tasks.

Use Spearman's rho or weighted Cohen's kappa for ordinal scales, precision/recall/F1 for binary classification, and agreement rate plus position consistency for pairwise comparisons. Target Spearman's rho above 0.8 for good human correlation.

Inconsistency comes from position bias, length bias, missing rubrics, and prompt sensitivity where minor wording changes cause 10-20% score swings. Mitigate with detailed rubrics, position swapping, and version-controlled evaluation prompts.

Use a Panel of LLMs (PoLL) for high-stakes decisions where individual model bias is unacceptable. Aggregate scores using the median across judges to reduce outlier influence, accepting higher cost for improved reliability.