evaluator-optimizer

Iteratively improve LLM output quality by running a generator-evaluator-optimizer loop that carries structured critique forward across rounds until a quality threshold is met. This loop applies to code, text, and plans where a rubric can score correctness.

Use MBR-style best-of-N selection to compare multiple candidates, score them against a rubric, and select the best using argmax or consensus voting. This ensures the highest quality output is chosen from parallel generations.

An LLM judge evaluates generated text by outputting structured evaluator verdicts containing a rubric-aligned score and blocking issues. These issues become direct optimizer context for the next generation round.

Yes, you need a stronger judge model to evaluate LLM generation quality accurately. Enforcing a judge capability guardrail ensures the evaluator is more capable than the evaluated model, preventing self-assessment bias.

Limitations of using an optimization loop include requiring explicit stopping criteria and a max-rounds budget to prevent infinite cycles. Additionally, the evaluator must produce structured verdicts with actionable blocking issues for effective threading.

Yes, you can use best-of-N selection for code generation and plan optimization. The evaluator-optimizer applies to any artifact where a rubric can score correctness, safety, accuracy, or compliance and best-of-N selection is needed.