skills.rest
kinnerkarmanish
kinnerkarmanishCOMMUNITY
Explore all skills by this author →
GitHub

validate-evaluator

Calibrate LLM judges against human labels and output a calibration report.

Updated Apr 26, 2026
Search Tags:#llm#metrics#calibration#confidence-intervals#bias-detection#evaluator#human-labels
One-click install
npx skills add https://github.com/kinnerkarmanish/mak --skill validate-evaluator-kinnerkarmanish
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: validate-evaluator
Source: https://github.com/kinnerkarmanish/mak/tree/main/library/skills/ai-patterns/validate-evaluator
Command: npx skills add https://github.com/kinnerkarmanish/mak --skill validate-evaluator-kinnerkarmanish

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Calibrates LLM judges against human labels with statistical rigor to ensure reliable automated evaluation.

Core Features & Use Cases

  • Compute classification metrics (TPR, TNR, precision, recall, F1) and Cohen's kappa to quantify agreement.
  • Detect systematic biases and provide actionable prompt-corrections.
  • Produce a calibration report with confidence intervals to guide deployment decisions.

Quick Start

Run the calibration workflow by providing a judge prompt and a human-labeled dataset to generate a reliability report.

Frequently Asked Questions about validate-evaluator

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calibrate an LLM evaluator against human labels?▼

To calibrate an LLM evaluator against human labels, provide a judge prompt and a human-labeled dataset to compute TPR, TNR, precision, recall, F1, and Cohen's kappa, generating a reliability report.

What metrics are used to measure LLM judge alignment with human labels?▼

LLM judge alignment with human labels is measured using classification metrics like TPR, TNR, precision, recall, F1, and Cohen's kappa, alongside systematic bias detection and confidence intervals.

How can I detect bias in my LLM evaluation prompts?▼

You can detect bias in LLM evaluation prompts by applying a calibration workflow that compares model judgments against human labels, diagnosing systematic biases and providing actionable prompt corrections.

How do I check if my automated LLM judge is statistically reliable?▼

You check automated LLM judge reliability by generating a calibration report that includes confidence intervals, Cohen's kappa, and classification metrics to statistically validate alignment with human labels.

Do I need a labeled dataset to diagnose LLM judge biases?▼

Yes, you need a human-labeled dataset to diagnose LLM judge biases, as the workflow compares the LLM evaluator's judgments against these human labels to quantify agreement and detect systematic biases.

Related Skills

Explore More Data & Analytics Skills→
Miosa-osaMiosa-osa
226

validate-evaluator

Calibrate LLM evaluators against human-labeled datasets and generate Markdown calibration reports.

Official
Advanced
ollieb89ollieb89

llm-evaluation

Evaluate LLM outputs with automated metrics and LLM-as-Judge workflows.

Community
Advanced
spandiosspandios

ab-test-analysis

Analyze A/B test results for statistical significance and actionable recommendations.

Community
Advanced
MantisWareMantisWare

validate-evaluator

Calibrate LLM-based evaluators against human labels and generate reliability reports.

Community
Advanced