validate-evaluator

Calibrate LLM judges against human labels using TPR/TNR measurements and bias correction.

5|Updated Oct 22, 2025

Search Tags:#llm evaluation #tnr #tpr #judge calibration #bias correction #human labels

Ready to run

npx skills add https://github.com/marchatton/agent-skills --skill validate-evaluator

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: validate-evaluator
Source: https://github.com/marchatton/agent-skills/tree/main/.agents/skills/08-evals/validate-evaluator
Command: npx skills add https://github.com/marchatton/agent-skills --skill validate-evaluator

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires scikit-learn, numpy, judgy, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill ensures that an AI judge's assessments align with human judgment, preventing biased or inaccurate AI evaluations before they impact production data.

Core Features & Use Cases

LLM Judge Calibration: Fine-tune LLM judges using human-labeled data to achieve high True Positive Rate (TPR) and True Negative Rate (TNR).
Bias Correction: Apply statistical methods to estimate the true success rate of an LLM judge on production data, accounting for known errors.
Use Case: After developing a prompt to evaluate user-generated content, use this Skill to test its accuracy against human moderators, ensuring it correctly identifies both good and bad content before deploying it.

Quick Start

Use the validate-evaluator skill to calibrate the judge prompt against the provided human-labeled dataset.

Frequently Asked Questions about validate-evaluator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I calibrate an LLM judge against human labels to ensure evaluation accuracy?▼

You calibrate an LLM judge against human labels by splitting binary Pass/Fail data, measuring True Positive Rate and True Negative Rate, and applying bias correction to ensure alignment before production deployment.

What is LLM judge calibration and when do I need it?▼

LLM judge calibration is the process of validating non-code-based evaluators against human-labeled data to ensure alignment and accuracy. You need it after prompt development to verify that the judge correctly identifies both good and bad content before trusting its outputs.

How do I correct bias in LLM evaluation outputs?▼

Correct bias in LLM evaluation outputs by applying statistical methods to estimate the true success rate on production data, accounting for known errors in True Positive Rate and True Negative Rate measurements derived from your human-labeled validation data.

Do I need human-labeled data with binary Pass/Fail labels to validate an evaluator?▼

Yes, validating an evaluator requires human-labeled data with binary Pass/Fail labels and candidate few-shot examples. This dataset is necessary to accurately measure TPR and TNR and to calibrate the LLM judge against human judgment.

Can I use scikit-learn and judgy for LLM judge calibration?▼

Yes, LLM judge calibration uses scikit-learn, numpy, and judgy as dependencies. These libraries support the statistical methods, data splitting, and bias correction needed to measure TPR and TNR against human labels.

What are the limitations of using TPR and TNR for LLM evaluation validation?▼

Using TPR and TNR for LLM evaluation validation requires binary Pass/Fail labels, meaning it does not support multi-class or scalar scoring. Additionally, calibration accuracy is strictly bounded by the quality and representativeness of your human-labeled dataset.