Audit zicato evaluation boards for harness correctness using canonical artifacts.
npx skills add https://github.com/pedapudi/zicato --skill zicato-audit-board
Please help me install this Agent Skill. Skill: zicato-audit-board Source: https://github.com/pedapudi/zicato/tree/main/skills/zicato-audit-board Command: npx skills add https://github.com/pedapudi/zicato --skill zicato-audit-board
This Skill detects evaluation-harness defects that can produce misleading zicato verdicts, including unwinnable ground truth, proxy-graded outputs, dead judges, inverted scoring, and nondeterministic results.
Use the zicato audit board skill to inspect the existing baseline run artifacts and determine whether the board can be trusted for tournament or evolution decisions.
High-intent search queries and answers about installing and using this skill.
You validate evaluation harness correctness by checking ground-truth winnability, graded-artifact fidelity, and judge-fire counts against baseline run artifacts. This verifies expectations behave as intended and catches unwinnable ground truth or dead judges.
Nondeterministic results and misleading verdicts stem from harness defects like proxy-graded outputs or inverted scoring. Inspect loss profiles, event streams, and board declarations to isolate structural evaluation problems before tournaments.
Ground-truth winnability checking verifies that trivially correct stubs pass and trivially wrong stubs fail. It confirms board expectations behave as intended, ensuring the evaluation harness can actually grade real agent outputs correctly.
You check if every declared judge fires by auditing judge-fire counts using canonical board and event artifacts. This harness mechanics audit confirms all declared judges execute as intended during the evaluation run.
Run an evaluation board audit after building or changing boards and whenever verdicts, pass rates, judge behavior, or candidate rankings appear suspicious. This ensures the board can be trusted for tournament or evolution decisions.
Verify pass-rate monotonicity and determinism by inspecting telemetry artifacts including loss profiles, generation scores, and event streams. This identifies nondeterministic results and inverted scoring, ensuring better candidates receive the intended scalar ranking.
pedapudiTrace zicato evaluation runs by correlating events.jsonl telemetry with loss.json profiles.
hamelsmuCalibrate LLM judges against human labels using TPR/TNR measurement and bias correction.
pedapudiTrace zicato inner-harness evolution across epochs and generations from lineage records.
nexus-labs-automationMeasure agent quality across automated evaluations, human feedback, and trend analysis.
