eval-calibration

LLM-as-judge evaluation systems suffer from a systematic generosity bias that inflates scores for flawed artifacts, failing to catch factual errors, missing citations, or structural gaps. Rubric calibration is required to eliminate this bias and produce trustworthy, production-ready quality scores.

Use a repeatable 6-step calibration process that builds a ground-truth catalogue of known flaws, tracks detection rates, runs multi-run variance testing, and checks fitness dimension coverage to ensure your rubric catches real issues and scores accurately.

Calibrated evaluation rubrics must meet an 80%+ ground-truth detection rate and maintain a score variance of 0.5 or less across runs. They must also demonstrate willingness to score flawed artifacts below 8 out of 10.

Apply built-in bias mitigation safeguards against score anchoring, missing fitness dimensions, and rubric inflation. These safeguards prevent false-pass quality gates by ensuring flawed app builds and chatbot transcripts are flagged accurately.

Yes, rubric calibration applies to any generated artifacts including app builds and chatbot transcripts. It calibrates per-skill evaluation rubrics against a ground-truth catalogue to ensure accurate scoring across all artifact types.

A ground-truth catalogue is a curated dataset of known artifact flaws used to calibrate LLM-as-judge rubrics. It enables detection rate tracking to verify the rubric catches factual, structural, and adversarial gaps before deployment.