skills.rest
Archive228
Archive228COMMUNITY
Explore all skills by this author →
GitHub

evaluator-calibration

Calibrate reviewer personas with few-shot rubric examples to prevent leniency drift.

746|130|Updated Jun 30, 2026
Search Tags:#AI evaluation#evaluator calibration#reproducible grading#reviewer consistency#rubric-based learning
One-click install
npx skills add https://github.com/Archive228/loopkit --skill evaluator-calibration
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluator-calibration
Source: https://github.com/Archive228/loopkit/tree/main/skills/evaluator-calibration
Command: npx skills add https://github.com/Archive228/loopkit --skill evaluator-calibration

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill helps maintain the consistency and accuracy of reviewer evaluations by calibrating the reviewer persona with few-shot rubric examples, preventing leniency drift over long runs.

Core Features & Use Cases

  • Calibration with Examples: Uses concrete pass/fail examples to anchor the rubric and prevent leniency drift.
  • Reproducible Verdicts: Ensures the same artifact evaluated by the same reviewer at different times yields the same verdict.
  • Use Case: When setting up a critic/evaluator/judge agent in a multi-agent loop, or when noticing evaluator scores drifting upward without a change in output quality.

Quick Start

Run the 'evaluator-calibration' skill with the provided examples and rubric to calibrate your reviewer agent.

Frequently Asked Questions about evaluator-calibration

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I prevent leniency drift in LLM evaluator agents during long grading runs?▼

Reproducible grading in multi-agent harnesses requires calibrating the reviewer persona with few-shot rubric examples and applying a consistent re-prompting cadence, ensuring the same artifact yields the same verdict across different times.

Why does my AI reviewer give inconsistent scores for the same output at different times?▼

AI reviewer inconsistency often occurs due to persona degradation over long runs; calibrating the reviewer with concrete pass/fail rubric examples anchors the evaluation criteria, preventing scores from drifting upward without actual quality changes.

What is the best way to set up a critic agent for reproducible verdicts in a multi-agent loop?▼

The best way to set up a critic agent for reproducible verdicts is providing a rubric with concrete pass/fail examples and maintaining a consistent re-prompting cadence to calibrate reviewer consistency throughout the loop.

Do I need concrete pass/fail examples to calibrate rubric-based learning for AI evaluation?▼

Yes, concrete pass/fail examples are required to calibrate rubric-based learning for AI evaluation, as they anchor the reviewer persona to prevent leniency drift and ensure consistent, reproducible grading verdicts over time.

Related Skills

Explore More Software Engineering Skills→
nthnclrknthnclrk
11

certification-program-builder

Define role-based readiness criteria and map them to certification artifacts.

Community
Advanced
Oruga420Oruga420

long-running-harness

Orchestrate multi-agent planning, generation, and evaluation loops with sprint contracts and file-based handoffs.

Community
Advanced
pedapudipedapudi
4

zicato-build-board

Design evaluation boards for multi-agent harnesses with judges and holdout splits.

Community
Advanced
kinnerkarmanishkinnerkarmanish

judge-prompt

Design binary pass/fail LLM-as-Judge evaluators with rubrics and guardrails.

Community
Advanced