write-judge-prompt

To evaluate subjective LLM outputs like tone or relevance, you can design a binary LLM-as-Judge evaluator that applies semantic assessment to subjective failure modes. This approach uses structured output definitions and few-shot examples to ensure consistent, actionable results.

Designing few-shot examples for an LLM-as-Judge prompt involves calibrating the evaluator with human-labeled trace examples of borderline cases. This few-shot calibration guides the judge to understand nuanced distinctions and apply consistent pass/fail criteria.

The best way to enforce structured output for an LLM-as-Judge evaluation is to standardize the response format to include both a detailed critique and a final binary verdict. This structured output enforcement ensures evaluation results are actionable and consistent.

Yes, you can use an LLM-as-Judge to evaluate if a customer support bot maintains a professional tone. You build a custom binary evaluator that detects specific failure modes, such as avoiding overly robotic language while maintaining professionalism.

You should not use binary classification for AI alignment evaluation when a subjective failure mode requires a spectrum of scores rather than a clear pass/fail verdict. This approach is strictly designed for binary evaluation, limiting its use for multi-tiered quality assurance.

Yes, you need human-labeled trace examples to build an LLM-as-Judge evaluator. Providing these examples during the design phase is required for few-shot calibration, ensuring the judge understands borderline cases and evaluates subjective failure modes accurately.