validate-evaluator

To calibrate an LLM judge against human labels, you need a labeled dataset and a candidate judge prompt. The process uses data splitting, TPR/TNR measurement, and bias correction to iteratively refine and validate alignment with human-defined Pass/Fail criteria.

TPR (True Positive Rate) and TNR (True Negative Rate) measurement in LLM evaluation quantifies judge accuracy by calculating how correctly the judge identifies positive and negative cases against human-labeled data, ensuring it accurately detects both good and bad responses.

You measure LLM judge bias on production data by applying statistical bias correction methods. This estimates the true success rate of the judge, preventing unreliable or skewed assessments when evaluating subjective quality criteria in a live environment.

Yes, you need labeled data to validate an LLM evaluator. Human-labeled examples are a strict requirement for the calibration process, acting as the ground truth to measure TPR/TNR performance and apply bias correction before production deployment.

You should use LLM judge calibration for data validation when evaluating subjective quality criteria, such as customer support responses. It ensures the judge correctly identifies both good and bad outputs before you trust the LLM evaluations in production.