Design outcome expectations and process judges for zicato board evaluations.
npx skills add https://github.com/pedapudi/zicato --skill zicato-design-judges
Please help me install this Agent Skill. Skill: zicato-design-judges Source: https://github.com/pedapudi/zicato/tree/main/skills/zicato-design-judges Command: npx skills add https://github.com/pedapudi/zicato --skill zicato-design-judges
This Skill helps define what a multi-agent system should be measured on, separating successful outcomes from the quality and safety of the process used to produce them.
Use this Skill to design outcome expectations and process judges for the behaviors your zicato board should measure.
High-intent search queries and answers about installing and using this skill.
Outcome expectations evaluate final deliverables using predicate, text, regex, JSON Schema, or rubric matchers. Process judges monitor live reasoning trajectories and tool-call behavior during execution, catching safety violations and drift as they happen rather than after completion.
You select matcher types—predicate, text, regex, JSON Schema, or rubric—based on your output format, then configure them on your evaluation board to validate final deliverables deterministically, separate from process quality scoring.
Yes, Python judges can parse structured tool-call events from telemetry ledgers to detect issues like fabricated metrics. Callable-separation safeguards preserve judge independence from the harness, ensuring you grade observable behavior rather than self-reported summaries.
Establish drift kinds, severity levels, and namespace weighting to balance outcome expectations with process monitoring. This scoring strategy defines how much each judge contributes to overall quality measurement and sets policies for judge-only runs.
Separating them prevents grading narration instead of observable behavior. Outcome expectations validate final deliverables deterministically, while independent process judges catch safety violations and drift from telemetry, ensuring neither masks the other's failures.
luicabref97Evaluate Claude Code agent outputs with structured rubrics and LLM-as-judge scoring.
pedapudiTrace zicato evaluation runs by correlating events.jsonl telemetry with loss.json profiles.
LayerDynamicsQuery Claude Code telemetry for tool call history, session stats, and error rates.
alex-voloshin-devValidate agent skills with linting and LLM-based calibration drift detection.
