Design evaluation boards for multi-agent harnesses with judges and holdout splits.
npx skills add https://github.com/pedapudi/zicato --skill zicato-build-board
Please help me install this Agent Skill. Skill: zicato-build-board Source: https://github.com/pedapudi/zicato/tree/main/skills/zicato-build-board Command: npx skills add https://github.com/pedapudi/zicato --skill zicato-build-board
This Skill helps operators turn vague quality goals into discriminating evaluation boards that reliably distinguish stronger and weaker multi-agent harnesses while avoiding overfitting and noisy signals.
Use the zicato board-building skill to draft and validate a holdout-aware evaluation board with entries, judges, and loss weights for the target agent behavior.
High-intent search queries and answers about installing and using this skill.
Evaluation boards prevent overfitting by reserving holdout splits for generalization checks, applying monotonicity protections to passing entries, and requiring explicit confirmation before rolling a new evaluation epoch.
Configure process judges by selecting telemetry sources and tuning drift, pass-rate, severity, per-kind, per-judge, runtime, and namespace weights to build a weighted loss profile that grades observable multi-agent behavior.
Validate evaluation board changes by building drafts, previewing validation warnings, configuring train and holdout slices, and applying the board only after explicit confirmation to prevent noisy signals.
Multi-agent evaluation board design requires explicit collusion prevention safeguards, observable telemetry grading, and validation checks to ensure process judges reliably distinguish stronger and weaker harnesses.
Measure final-answer correctness and tool-use quality by creating board entries with expectations, tags, and weights, then assigning process judges to monitor off-topic drift and tool telemetry.
You can combine single-turn, scripted multi-turn, and emulated multi-turn tasks in the same evaluation board by assigning each entry specific expectations, tags, weights, and holdout coverage.
marmikcfcDefine multi-email sequences with timing, subject lines, and success metrics.
pedapudiDesigns zicato evaluation boards that distinguish improved harnesses from challengers.
pedapudiDesign outcome expectations and process judges for zicato board evaluations.
flutterOrchestrate automated evaluation of agent skills using isolated subagents and rubrics.
