write-judge-prompt

An LLM-as-judge prompt for binary classification requires four components: a task criterion, explicit Pass/Fail definitions, few-shot examples, and structured JSON output. This approach evaluates exactly one targeted failure mode per judge to keep verdicts actionable.

Evaluating tone mismatch without ordinal scoring requires a binary LLM-as-judge prompt that enforces strict Pass/Fail verdicts. You build the judge prompt using labeled Pass, Fail, and borderline examples from your training split to reduce ambiguity and output a structured JSON format containing both a critique and a final result.

To calibrate an LLM evaluation prompt with few-shot examples, provide labeled Pass, Fail, and borderline examples from your training data split. This few-shot calibration reduces ambiguity and helps the binary judge consistently output a structured JSON verdict with a detailed critique.

You should use a binary LLM judge instead of Likert scoring because a narrow, actionable judge evaluating a single failure mode produces clearer results. Binary Pass/Fail output with explicit definitions prevents the ambiguity associated with ordinal scoring in evaluation pipelines.

You cannot evaluate multiple failure modes in one prompt with this approach. The judge is designed to evaluate exactly one targeted failure mode per prompt to keep verdicts actionable, requiring a task criterion, Pass/Fail definitions, few-shot examples, and structured JSON output.

An LLM-as-judge binary classification prompt requires structured JSON output containing both a critique and a final verdict. This structured format ensures the evaluation pipeline receives consistent, machine-readable results instead of unstructured text.