simmer-judge-board

Panel-based evaluation reduces bias by dispatching multiple judges with distinct lenses to independently score a candidate, deliberate on differences, and converge on consensus scores, surfacing blind spots that a single judge might miss.

The best way to run consensus scoring is to dispatch a panel of three independent judges to evaluate the candidate in parallel, enforce one deliberation round to challenge scores, and synthesize a unified report with consensus scores and a single ASI.

Yes, you can use a judge board for prompt evaluation because it outputs a unified report that preserves the standard single-judge format, ensuring the orchestrator receives indistinguishable results while gaining diverse perspectives.

Multi-judge deliberation works with iterative code refinement by applying diverse perspectives to challenge scores and surface blind spots, driving robust improvements across text, code, or pipelines where stagnation is a risk.

You need a multi-judge board when iterative evaluation contexts require multiple lenses to surface blind spots and drive robust improvements, especially when bias or stagnation is a risk in text, code, or pipeline refinement.

A limitation of the deliberation board is that it enforces exactly one round of deliberation and cannot be invoked directly by users; it must be dispatched by the simmer orchestrator in board mode.