validate-evaluator

Calibrate LLM judges against human labels using TPR/TNR measurement and bias correction.

1.6k|164|Updated Mar 1, 2026

Search Tags:#llm evaluation #data validation #tnr #tpr #judge calibration #bias correction

Ready to run

npx skills add https://github.com/hamelsmu/evals-skills --skill validate-evaluator-hamelsmu

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: validate-evaluator
Source: https://github.com/hamelsmu/evals-skills/tree/main/skills/validate-evaluator
Command: npx skills add https://github.com/hamelsmu/evals-skills --skill validate-evaluator-hamelsmu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires sklearn, numpy, judgy, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill ensures that your LLM-based judges are accurately evaluating outputs by calibrating them against human judgment, preventing biased or unreliable assessments.

Core Features & Use Cases

LLM Judge Calibration: Fine-tune LLM judges to align with human-defined Pass/Fail criteria.
Performance Measurement: Quantify judge accuracy using True Positive Rate (TPR) and True Negative Rate (TNR).
Bias Correction: Apply statistical methods to estimate the true success rate of the judge on production data.
Use Case: After developing a judge prompt to evaluate customer support responses, use this skill to test its performance against human-labeled examples, ensuring it correctly identifies both good and bad responses before deploying it.

Quick Start

Use the validate-evaluator skill to calibrate the LLM judge against the provided human-labeled dataset.

Frequently Asked Questions about validate-evaluator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I calibrate an LLM judge against human labels?▼

To calibrate an LLM judge against human labels, you need a labeled dataset and a candidate judge prompt. The process uses data splitting, TPR/TNR measurement, and bias correction to iteratively refine and validate alignment with human-defined Pass/Fail criteria.

What is TPR and TNR measurement in LLM evaluation?▼

TPR (True Positive Rate) and TNR (True Negative Rate) measurement in LLM evaluation quantifies judge accuracy by calculating how correctly the judge identifies positive and negative cases against human-labeled data, ensuring it accurately detects both good and bad responses.

How do I measure LLM judge bias on production data?▼

You measure LLM judge bias on production data by applying statistical bias correction methods. This estimates the true success rate of the judge, preventing unreliable or skewed assessments when evaluating subjective quality criteria in a live environment.

Do I need labeled data to validate an LLM evaluator?▼

Yes, you need labeled data to validate an LLM evaluator. Human-labeled examples are a strict requirement for the calibration process, acting as the ground truth to measure TPR/TNR performance and apply bias correction before production deployment.

When should I use LLM judge calibration for data validation?▼

You should use LLM judge calibration for data validation when evaluating subjective quality criteria, such as customer support responses. It ensures the judge correctly identifies both good and bad outputs before you trust the LLM evaluations in production.