write-judge-prompt

Name: write-judge-prompt
Availability: InStock
Author: majidraza1228

Design binary Pass/Fail LLM-as-Judge prompts for defined failure modes.

Updated Apr 19, 2026

Search Tags:#error-analysis #llm-evaluation #judge-prompt #binary-classification #trace-labeling #criteria-definition

Ready to run

npx skills add https://github.com/majidraza1228/eval-framework --skill write-judge-prompt-majidraza1228

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: write-judge-prompt
Source: https://github.com/majidraza1228/eval-framework/tree/main/skills/write-judge-prompt
Command: npx skills add https://github.com/majidraza1228/eval-framework --skill write-judge-prompt-majidraza1228

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you create consistent, binary Pass/Fail LLM-as-Judge prompts for one clearly defined failure mode, enabling objective evaluation where subjective judgments are unreliable.

Core Features & Use Cases

Four-component judge prompts: Task & Evaluation Criterion, Pass/Fail Definitions, Few-Shot Examples, and a Structured Output Format.
Uses human-labeled traces (minimum 20 Pass and 20 Fail) to ground the judge in real-world data.
Applies when a code-based evaluator cannot reliably assess the failure mode, such as tone, faithfulness, or interpretation issues.

Quick Start

Provide a failure-mode description and a set of human-labeled traces to generate the four-component judge prompt.

Frequently Asked Questions about write-judge-prompt

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I create a binary LLM-as-Judge prompt for evaluating a single failure mode?▼

To create a binary LLM-as-Judge prompt, define a single failure mode and use human-labeled traces to build a four-component prompt: Task & Evaluation Criterion, Pass/Fail Definitions, Few-Shot Examples, and a Structured Output Format. This enforces objective pass/fail evaluation.

What is binary classification in LLM evaluation and when should I use it?▼

Binary classification in LLM evaluation assesses outputs using strict pass/fail criteria for one defined failure mode. Use it when code-based evaluators cannot reliably assess subjective qualities like tone, faithfulness, or reasoning quality, ensuring consistent and structured judgments.

How many human-labeled traces do I need to build a reliable judge prompt?▼

You need a minimum of 40 human-labeled traces to build a reliable judge prompt, comprising at least 20 Pass examples and 20 Fail examples. These traces ground the LLM-as-Judge in real-world data for accurate binary classification.

Does LLM-as-Judge evaluation work for subjective criteria like tone and faithfulness?▼

LLM-as-Judge evaluation works for subjective criteria like tone and faithfulness by applying strict binary definitions and structured output formats. It replaces unreliable code-based evaluators with grounded few-shot examples from human-labeled traces.

What's the best way to structure an LLM evaluation prompt for error analysis?▼

The best way to structure an LLM evaluation prompt is using four required components: Task & Evaluation Criterion, Pass/Fail Definitions, Few-Shot Examples, and a Structured Output Format. This design relies on a pinned model version and a defined evidence subset for reliable trace labeling.

Why should I limit my judge prompt to a single failure mode instead of multiple issues?▼

Limiting your judge prompt to a single failure mode ensures objective binary classification by enforcing one clear pass/fail outcome. Evaluating multiple issues simultaneously dilutes criteria definition and reduces the accuracy of error-analysis-defined cases like reasoning quality.