meta-debug

What problem does it solve?

This Skill detects inconsistencies across an agent's prompt, ideal state, and evaluation rubrics to avoid wasted optimization cycles and ensure that automatic evaluation reflects true user value.

Core Features & Use Cases

• Evidence-based per-case diagnostics: Extracts scores, dimension breakdowns, and judge comments from eval results and ties each diagnostic to concrete rubric, prompt, or testcase excerpts.
• Cross-case pattern detection: Computes score distributions and identifies systemic issues such as consistently low dimensions, over-triggered negative criteria, and rubric coverage gaps.
• Decision item generation for HCI review: Produces user-facing decisions (with skip option) and auto-applicable actions where safe, exporting a strict calibration_report.json consumable by the calibration viewer.
• Use Cases: Initial calibration after agent creation, first-run validation before iterative tuning, and pre-release sanity checks for evaluation fairness.

Quick Start

Calibrate the agent 'cls-log-agent' by providing its prompt, ideal_state, testcases, evaluation results directory, and an output directory to generate calibration_report.json.