validate-evaluator

To calibrate an LLM evaluator against human labels, you validate your judge prompt using a human-labeled dataset to compute statistical metrics, run bias checks, and generate a reliability calibration report.

Evaluating LLM judge reliability involves computing statistical metrics such as TPR, TNR, precision, recall, F1 score, and Cohen's kappa to measure agreement between the evaluator and human labels.

Yes, you can detect systematic biases in LLM evaluator prompts by running bias checks against a human-labeled dataset, which generates actionable correction recommendations for model prompts.

Yes, validating an LLM evaluator requires a human-labeled dataset and a judge prompt to compute metrics and generate a calibration report, with an optional bootstrap configuration for enhanced analysis.

The best way to validate judge prompts for classification tasks is to apply them to a human-labeled dataset, computing metrics like precision and recall, and outputting a Markdown calibration report with bias findings.

A low Cohen's kappa in an LLM evaluator calibration report indicates poor agreement between the evaluator and human labels, highlighting areas where the judge prompt requires correction or bias mitigation.