validate-evaluator

Name: validate-evaluator
Availability: InStock
Author: kinnerkarmanish

Calibrate LLM judges against human labels and output a calibration report.

Updated Apr 26, 2026

Search Tags:#llm #metrics #calibration #confidence-intervals #bias-detection #evaluator #human-labels

Ready to run

npx skills add https://github.com/kinnerkarmanish/mak --skill validate-evaluator-kinnerkarmanish

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: validate-evaluator
Source: https://github.com/kinnerkarmanish/mak/tree/main/library/skills/ai-patterns/validate-evaluator
Command: npx skills add https://github.com/kinnerkarmanish/mak --skill validate-evaluator-kinnerkarmanish

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Calibrates LLM judges against human labels with statistical rigor to ensure reliable automated evaluation.

Core Features & Use Cases

Compute classification metrics (TPR, TNR, precision, recall, F1) and Cohen's kappa to quantify agreement.
Detect systematic biases and provide actionable prompt-corrections.
Produce a calibration report with confidence intervals to guide deployment decisions.

Quick Start

Run the calibration workflow by providing a judge prompt and a human-labeled dataset to generate a reliability report.

Frequently Asked Questions about validate-evaluator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I calibrate an LLM evaluator against human labels?▼

To calibrate an LLM evaluator against human labels, provide a judge prompt and a human-labeled dataset to compute TPR, TNR, precision, recall, F1, and Cohen's kappa, generating a reliability report.

What metrics are used to measure LLM judge alignment with human labels?▼

LLM judge alignment with human labels is measured using classification metrics like TPR, TNR, precision, recall, F1, and Cohen's kappa, alongside systematic bias detection and confidence intervals.

How can I detect bias in my LLM evaluation prompts?▼

You can detect bias in LLM evaluation prompts by applying a calibration workflow that compares model judgments against human labels, diagnosing systematic biases and providing actionable prompt corrections.

How do I check if my automated LLM judge is statistically reliable?▼

You check automated LLM judge reliability by generating a calibration report that includes confidence intervals, Cohen's kappa, and classification metrics to statistically validate alignment with human labels.

Do I need a labeled dataset to diagnose LLM judge biases?▼

Yes, you need a human-labeled dataset to diagnose LLM judge biases, as the workflow compares the LLM evaluator's judgments against these human labels to quantify agreement and detect systematic biases.