skills.rest
adamatdevops
adamatdevopsCOMMUNITY
Explore all skills by this author →
GitHub

validate-evaluator

Calibrate an LLM judge against human Pass/Fail labels using TPR/TNR metrics.

1|Updated Jan 7, 2026
Search Tags:#llm evaluation#tnr#bias correction#bootstrap ci#tp r#rogan-gladen#prompt calibration
One-click install
npx skills add https://github.com/adamatdevops/forge-works --skill validate-evaluator-adamatdevops
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: validate-evaluator
Source: https://github.com/adamatdevops/forge-works/tree/main/.skills/hamelsmu/validate-evaluator
Command: npx skills add https://github.com/adamatdevops/forge-works --skill validate-evaluator-adamatdevops

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

It prevents unreliable automated pass/fail judgments by calibrating an LLM judge against human labels and measuring its bias-correctable performance.

Core Features & Use Cases

  • TPR/TNR-based calibration: Optimizes a judge specifically using True Positive Rate and True Negative Rate rather than accuracy or precision/recall.
  • Train/dev/test discipline: Separates few-shot training examples from a development set used for iterative refinement and a held-out test set used once for final measurement.
  • Bias correction for production: Uses Rogan-Gladen correction to estimate true success rates on unlabeled production data.
  • Uncertainty quantification: Produces a bootstrap confidence interval so stakeholders understand estimation reliability.

Quick Start

Calibrate the judge for your binary failure mode by running the steps to split ~100 labeled examples, measure TPR/TNR on dev, and then measure TPR/TNR once on the held-out test set.

Frequently Asked Questions about validate-evaluator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calibrate an LLM judge using TPR and TNR metrics?▼

Calibrate an LLM judge against human binary Pass/Fail labels by measuring and optimizing True Positive Rate and True Negative Rate using disjoint train/dev/test splits and iterative prompt refinement.

How do I correct LLM evaluation bias for production success rates?▼

Correct LLM evaluation bias for production success rates by applying Rogan-Gladen correction to estimate true success rates on unlabeled production data based on measured judge performance.

What is the best way to measure uncertainty in LLM judge calibration?▼

Measure uncertainty in LLM judge calibration by generating a bootstrap confidence interval, allowing stakeholders to understand the reliability of the bias-corrected production success rate estimations.

Do I need human-labeled data to evaluate LLM judge performance?▼

Yes, evaluating LLM judge performance requires human-labeled expert data to serve as the ground truth for calibrating True Positive Rate and True Negative Rate across train, dev, and test splits.

Why use TPR and TNR instead of accuracy for LLM evaluation?▼

Use TPR and TNR instead of accuracy to optimize an LLM judge for specific binary failure modes, ensuring strict metric rules that prevent unreliable automated pass/fail judgments.

How do I split data for iterative prompt refinement in LLM calibration?▼

Split approximately 100 labeled examples into disjoint train, development, and test sets; use train for few-shot examples, dev for iterative refinement, and test once for final measurement.

Related Skills

Explore More Data & Analytics Skills→
majidraza1228majidraza1228

write-judge-prompt

Design binary Pass/Fail LLM-as-Judge prompts for defined failure modes.

Community
Advanced
maragudkmaragudk
10

llm-as-a-judge

Build and validate LLM-as-Judge evaluators for automated quality assessment.

Official
Advanced
hamelsmuhamelsmu
1.6k

validate-evaluator

Calibrate LLM judges against human labels using TPR/TNR measurement and bias correction.

Community
Advanced
Miosa-osaMiosa-osa
226

validate-evaluator

Calibrate LLM evaluators against human-labeled datasets and generate Markdown calibration reports.

Official
Advanced