validate-evaluator

You calibrate an LLM judge against human labels by splitting binary Pass/Fail data, measuring True Positive Rate and True Negative Rate, and applying bias correction to ensure alignment before production deployment.

LLM judge calibration is the process of validating non-code-based evaluators against human-labeled data to ensure alignment and accuracy. You need it after prompt development to verify that the judge correctly identifies both good and bad content before trusting its outputs.

Correct bias in LLM evaluation outputs by applying statistical methods to estimate the true success rate on production data, accounting for known errors in True Positive Rate and True Negative Rate measurements derived from your human-labeled validation data.

Yes, validating an evaluator requires human-labeled data with binary Pass/Fail labels and candidate few-shot examples. This dataset is necessary to accurately measure TPR and TNR and to calibrate the LLM judge against human judgment.

Yes, LLM judge calibration uses scikit-learn, numpy, and judgy as dependencies. These libraries support the statistical methods, data splitting, and bias correction needed to measure TPR and TNR against human labels.

Using TPR and TNR for LLM evaluation validation requires binary Pass/Fail labels, meaning it does not support multi-class or scalar scoring. Additionally, calibration accuracy is strictly bounded by the quality and representativeness of your human-labeled dataset.