write-judge-prompt

To create a binary LLM-as-Judge prompt, define a single failure mode and use human-labeled traces to build a four-component prompt: Task & Evaluation Criterion, Pass/Fail Definitions, Few-Shot Examples, and a Structured Output Format. This enforces objective pass/fail evaluation.

Binary classification in LLM evaluation assesses outputs using strict pass/fail criteria for one defined failure mode. Use it when code-based evaluators cannot reliably assess subjective qualities like tone, faithfulness, or reasoning quality, ensuring consistent and structured judgments.

You need a minimum of 40 human-labeled traces to build a reliable judge prompt, comprising at least 20 Pass examples and 20 Fail examples. These traces ground the LLM-as-Judge in real-world data for accurate binary classification.

LLM-as-Judge evaluation works for subjective criteria like tone and faithfulness by applying strict binary definitions and structured output formats. It replaces unreliable code-based evaluators with grounded few-shot examples from human-labeled traces.

The best way to structure an LLM evaluation prompt is using four required components: Task & Evaluation Criterion, Pass/Fail Definitions, Few-Shot Examples, and a Structured Output Format. This design relies on a pinned model version and a defined evidence subset for reliable trace labeling.

Limiting your judge prompt to a single failure mode ensures objective binary classification by enforcing one clear pass/fail outcome. Evaluating multiple issues simultaneously dilutes criteria definition and reduces the accuracy of error-analysis-defined cases like reasoning quality.