Build a binary LLM evaluator that returns JSON Pass or Fail verdicts.
npx skills add https://github.com/breethomas/bette-think --skill build-judge
Please help me install this Agent Skill. Skill: build-judge Source: https://github.com/breethomas/bette-think/tree/main/plugins/pm-thought-partner/skills/build-judge Command: npx skills add https://github.com/breethomas/bette-think --skill build-judge
This Skill helps product teams create a focused, repeatable LLM-based evaluator that issues a binary Pass or Fail for one specific failure mode that cannot be reliably checked with code, reducing manual labeling drift and increasing evaluation consistency.
Build a pass/fail LLM judge for the identified failure mode using 20+ labeled pass and fail examples and return a JSON critique and Pass or Fail verdict.
High-intent search queries and answers about installing and using this skill.
Build an LLM judge by creating a focused evaluator that issues a binary Pass or Fail verdict for one specific failure mode using few-shot examples of clear and borderline cases, requiring structured JSON output with a critique string and result.
A binary LLM judge is an automated evaluator that decides Pass or Fail for generated outputs when subjective failures like tone, faithfulness, relevance, or completeness cannot be checked with code and require human-labeled examples for validation.
Create a few-shot LLM evaluator using 20 or more human-labeled pass and fail examples, including clear Pass, clear Fail, and borderline cases, to teach the judge nuance before testing against held-out labels.
No, this approach requires building one judge per failure mode, such as tone mismatch, faithfulness, relevance, completeness, or instruction following, to ensure focused and repeatable evaluation consistency.
Validate an LLM judge by testing its structured critique and Pass or Fail verdicts against held-out human-labeled examples to measure agreement and reduce manual labeling drift before deployment.
Use a binary LLM judge when subjective failure modes like tone, faithfulness, relevance, or completeness cannot be reliably checked with code, requiring human-labeled examples and structured JSON output for consistent evaluation.
adamatdevopsGenerate binary Pass/Fail LLM-as-judge prompts with JSON critique output.
Miosa-osaDesign binary pass/fail LLM-as-Judge evaluation prompts with scoring instructions and guardrails.
kinnerkarmanishDesign binary pass/fail LLM-as-Judge evaluators with rubrics and guardrails.
marchattonDesign LLM-as-Judge prompts for binary Pass/Fail subjective evaluations.
