write-judge-prompt

Design LLM-as-Judge prompts for binary Pass/Fail subjective evaluations.

5|Updated Oct 22, 2025

Search Tags:#testing #prompt engineering #ai #llm #evaluation #judge

Ready to run

npx skills add https://github.com/marchatton/agent-skills --skill write-judge-prompt

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: write-judge-prompt
Source: https://github.com/marchatton/agent-skills/tree/main/.agents/skills/08-evals/write-judge-prompt
Command: npx skills add https://github.com/marchatton/agent-skills --skill write-judge-prompt

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you create reliable LLM-as-Judge evaluators for subjective criteria that are difficult or impossible to check with automated code-based methods.

Core Features & Use Cases

Subjective Evaluation: Design prompts for LLMs to act as judges for nuanced criteria like tone, faithfulness, or relevance.
Binary Pass/Fail: Focuses on creating clear, binary Pass/Fail judgments for specific failure modes.
Use Case: You need to evaluate if customer support responses maintain a consistently empathetic tone. This Skill guides you in creating a judge prompt to assess that specific criterion.

Quick Start

Use the write-judge-prompt skill to design a judge for evaluating email tone.

Frequently Asked Questions about write-judge-prompt

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate subjective LLM outputs like tone and relevance?▼

To evaluate subjective LLM outputs like tone and relevance, you design an LLM-as-Judge evaluator with clear definitions and few-shot examples. This approach handles nuanced criteria that automated code-based checks cannot interpret.

What is an LLM-as-Judge evaluator used for?▼

An LLM-as-Judge evaluator is used for assessing subjective failure modes requiring interpretation, such as faithfulness or tone. It provides binary Pass/Fail assessments using structured output formats instead of manual review.

How do I create a binary Pass/Fail judge prompt for AI testing?▼

You create a binary Pass/Fail judge prompt by defining clear evaluation criteria, supplying few-shot examples, and specifying a structured output format. This guides the LLM to produce consistent binary assessments for specific failure modes.

When should I use an LLM-as-Judge instead of code-based checks?▼

Use an LLM-as-Judge instead of code-based checks when a failure mode requires interpretation, such as evaluating empathetic tone or relevance. If the criteria cannot be validated through automated code, a judge evaluator is necessary.

Can I use this approach to validate or calibrate my existing judge models?▼

No, this approach designs new LLM-as-Judge evaluators for subjective criteria but does not validate or calibrate existing judge models. It focuses on generating structured prompts for binary Pass/Fail assessments.