advanced-evaluation

Structure the pipeline as criteria loading, primary scoring, bias mitigation, and confidence calibration. Use direct scoring for objective criteria and pairwise comparison for subjective judgments, always requiring evidence before scores.

Run the comparison twice with swapped response positions, then check consistency. If both passes agree, average their confidences; if they disagree, return a tie with reduced confidence since position bias influenced the result.

Use direct scoring when objective ground truth exists, such as factual accuracy or instruction following. Use pairwise comparison for subjective preferences like tone, style, or persuasiveness, where it correlates better with human judgment.

Use Spearman's rho or Kendall's tau for ordinal scores and Cohen's kappa for categorical judgments. Values above 0.8 for rho or 0.7 for kappa indicate strong agreement; also monitor position consistency and length-score correlation for bias.

Inconsistency usually comes from missing rubrics, overloaded criteria measuring multiple things, or sensitivity to prompt wording. Fix it by generating rubrics with explicit level descriptions and edge cases, and version-controlling evaluation prompts.

Use a different model family for evaluation than for generation, since models rate their own outputs higher. Anonymizing responses by removing model-identifying phrases further reduces this bias.