evaluator-calibration

What problem does it solve?

This Skill helps maintain the consistency and accuracy of reviewer evaluations by calibrating the reviewer persona with few-shot rubric examples, preventing leniency drift over long runs.

Core Features & Use Cases

• Calibration with Examples: Uses concrete pass/fail examples to anchor the rubric and prevent leniency drift.
• Reproducible Verdicts: Ensures the same artifact evaluated by the same reviewer at different times yields the same verdict.
• Use Case: When setting up a critic/evaluator/judge agent in a multi-agent loop, or when noticing evaluator scores drifting upward without a change in output quality.

Quick Start

Run the 'evaluator-calibration' skill with the provided examples and rubric to calibrate your reviewer agent.