Design binary pass/fail LLM-as-Judge evaluation prompts with scoring instructions and guardrails.
npx skills add https://github.com/Miosa-osa/canopy --skill judge-prompt
Please help me install this Agent Skill. Skill: judge-prompt Source: https://github.com/Miosa-osa/canopy/tree/main/library/skills/ai-patterns/judge-prompt Command: npx skills add https://github.com/Miosa-osa/canopy --skill judge-prompt
Design binary pass/fail evaluation prompts for LLMs to standardize judgments of model outputs and reduce subjective variability.
Invoke the judge-prompt CLI with a specific task to generate a ready-to-deploy evaluation prompt for your pipeline.
High-intent search queries and answers about installing and using this skill.
Binary LLM-as-Judge evaluation prompts standardize pass/fail judgments of model outputs to reduce subjective variability. They define explicit criteria, rubrics, and guardrails within evaluation pipelines to ensure consistent scoring.
Design an LLM evaluation rubric by defining concrete binary pass/fail criteria and explicit edge case descriptions. Few-shot calibration examples are then selected to stabilize model judgments and mitigate bias during evaluation.
Mitigate bias in LLM-as-Judge prompts by incorporating built-in guardrails directly into the scoring instructions. These guardrails counter common evaluation biases and are applied alongside few-shot calibration to stabilize pass/fail judgments.
Yes, you can generate judge prompts for multiple evaluation contexts by invoking the CLI for interactive design. It produces ready-to-deploy prompts with scoring instructions and guardrails tailored to specific evaluation pipelines.
LLM evaluation prompts produce inconsistent pass/fail judgments due to subjective variability and missing few-shot calibration. Adding explicit rubrics, edge case descriptions, and bias mitigation guardrails stabilizes the model's scoring behavior.
coval-aiAnalyze human versus LLM judge annotation discrepancies to refine evaluation prompts.
wangyouanEvaluate LLM outputs with direct scoring, pairwise comparison, and bias mitigation.
slowman2084Generate ten YAML test cases with empty ExpectedOutput and Judge fields.
ChakshuGautamEvaluate agent outputs against multi-dimensional rubrics to produce scores and pass/fail judgments.
