write-judge-prompt

Name: write-judge-prompt
Availability: InStock
Author: adamatdevops

Generate binary Pass/Fail LLM-as-judge prompts with JSON critique output.

1|Updated Jan 7, 2026

Search Tags:#prompt engineering #llm-as-judge #evaluation #structured output #few-shot examples #binary classification

Ready to run

npx skills add https://github.com/adamatdevops/forge-works --skill write-judge-prompt-adamatdevops

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: write-judge-prompt
Source: https://github.com/adamatdevops/forge-works/tree/main/.skills/hamelsmu/write-judge-prompt
Command: npx skills add https://github.com/adamatdevops/forge-works --skill write-judge-prompt-adamatdevops

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you turn a single, known failure mode into a precise LLM-as-judge prompt that outputs only a binary Pass/Fail verdict with a detailed critique.

Core Features & Use Cases

Single failure-mode judging: Evaluates exactly one targeted failure mode per judge to keep verdicts actionable.
Binary, non-ordinal results: Enforces strict Pass/Fail output (no Likert or scores) with clear definitions.
Few-shot calibration: Builds the judge prompt using labeled Pass, Fail, and borderline examples from the training split to reduce ambiguity.
Structured output: Requires a JSON format containing both a critique and a final result.

Quick Start

Ask the AI to generate a judge prompt that evaluates whether a generated response mismatches the required client persona tone, using your provided Pass, Fail, and borderline examples.

Frequently Asked Questions about write-judge-prompt

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I create an LLM-as-judge prompt for binary classification of a single failure mode?▼

An LLM-as-judge prompt for binary classification requires four components: a task criterion, explicit Pass/Fail definitions, few-shot examples, and structured JSON output. This approach evaluates exactly one targeted failure mode per judge to keep verdicts actionable.

What is the best way to evaluate tone mismatch in LLM responses without using ordinal scoring?▼

Evaluating tone mismatch without ordinal scoring requires a binary LLM-as-judge prompt that enforces strict Pass/Fail verdicts. You build the judge prompt using labeled Pass, Fail, and borderline examples from your training split to reduce ambiguity and output a structured JSON format containing both a critique and a final result.

How do I use few-shot examples to calibrate an LLM evaluation prompt for Pass/Fail verdicts?▼

To calibrate an LLM evaluation prompt with few-shot examples, provide labeled Pass, Fail, and borderline examples from your training data split. This few-shot calibration reduces ambiguity and helps the binary judge consistently output a structured JSON verdict with a detailed critique.

Why should I use a binary LLM judge instead of Likert scoring for my evaluation pipeline?▼

You should use a binary LLM judge instead of Likert scoring because a narrow, actionable judge evaluating a single failure mode produces clearer results. Binary Pass/Fail output with explicit definitions prevents the ambiguity associated with ordinal scoring in evaluation pipelines.

Can I evaluate multiple failure modes in one LLM-as-judge prompt?▼

You cannot evaluate multiple failure modes in one prompt with this approach. The judge is designed to evaluate exactly one targeted failure mode per prompt to keep verdicts actionable, requiring a task criterion, Pass/Fail definitions, few-shot examples, and structured JSON output.

Does an LLM-as-judge binary classification prompt require structured JSON output?▼

An LLM-as-judge binary classification prompt requires structured JSON output containing both a critique and a final verdict. This structured format ensures the evaluation pipeline receives consistent, machine-readable results instead of unstructured text.