write-judge-prompt

To evaluate subjective LLM outputs like tone and relevance, you design an LLM-as-Judge evaluator with clear definitions and few-shot examples. This approach handles nuanced criteria that automated code-based checks cannot interpret.

An LLM-as-Judge evaluator is used for assessing subjective failure modes requiring interpretation, such as faithfulness or tone. It provides binary Pass/Fail assessments using structured output formats instead of manual review.

You create a binary Pass/Fail judge prompt by defining clear evaluation criteria, supplying few-shot examples, and specifying a structured output format. This guides the LLM to produce consistent binary assessments for specific failure modes.

Use an LLM-as-Judge instead of code-based checks when a failure mode requires interpretation, such as evaluating empathetic tone or relevance. If the criteria cannot be validated through automated code, a judge evaluator is necessary.

No, this approach designs new LLM-as-Judge evaluators for subjective criteria but does not validate or calibrate existing judge models. It focuses on generating structured prompts for binary Pass/Fail assessments.