advanced-evaluation

Define clear criteria with weights, build a scoring prompt requiring evidence-based justification before scores, and parse structured JSON output. Use direct scoring for objective criteria like accuracy and pairwise comparison for subjective preferences like tone.

Run the comparison twice with swapped response positions, then check consistency. If both passes agree on the winner, average the confidences; if they disagree, return a tie with reduced confidence since position bias influenced the result.

Use direct scoring when objective ground truth exists, such as factual accuracy or instruction following. Use pairwise comparison for subjective preference judgments like tone, style, or persuasiveness, where it achieves higher agreement with human judges.

Use Spearman's rho or Kendall's tau for ordinal score correlation, Cohen's kappa for categorical judgments, and agreement rate plus position consistency for pairwise comparisons. Values above 0.8 for Spearman's rho indicate strong alignment.

Inconsistency stems from position bias, length bias, vague criteria, and missing rubrics. Fix this by requiring justification before scores, swapping positions in comparisons, and generating detailed rubrics with level descriptions and edge cases.