meta-debug

To diagnose evaluation inconsistencies, this Skill cross-references prompt definitions, ideal states, and rubric annotations to detect systemic issues and output a calibration_report.json. It extracts per-case evidence and computes score distributions to identify over-triggered negative criteria.

Generating a calibration report requires providing testcases, evaluation results, and rubric annotations to compute score distributions and cross-case patterns. The Skill processes these inputs to output a strict calibration_report.json containing structured diagnostics and user-facing decision items.

Agent evaluation calibration is the process of validating that automatic evaluation reflects true user value by checking consistency across prompts, ideal states, and rubrics. You need it during initial agent creation, first-run validation, and pre-release sanity checks for evaluation fairness.

This diagnostic tool is designed strictly for single-agent evaluation sessions with testcases, eval results, and rubric annotations. It computes per-case evidence and decision options specifically for individual agents rather than multi-agent orchestration workflows.

Running agent evaluation diagnostics requires evaluation markdowns, actual outputs, and rubric annotations as inputs. The Skill reads these formats to extract scores, dimension breakdowns, and judge comments, exporting results as a structured calibration_report.json.

Inconsistent rubric scores often stem from misalignment between the agent prompt, ideal state, and evaluation rubrics. This Skill detects these inconsistencies by computing score distributions and identifying systemic issues like over-triggered negative criteria or rubric coverage gaps.