Build and validate LLM-as-Judge evaluators for automated quality assessment.
npx skills add https://github.com/maragudk/evals-skills --skill llm-as-a-judge
Please help me install this Agent Skill. Skill: llm-as-a-judge Source: https://github.com/maragudk/evals-skills/tree/main/skills/llm-as-a-judge Command: npx skills add https://github.com/maragudk/evals-skills --skill llm-as-a-judge
This Skill automates the quality assessment of LLM pipeline outputs by using another LLM as a judge, enabling objective evaluation of nuanced or subjective failure modes.
Use the llm-as-a-judge skill to create a judge prompt for evaluating response helpfulness.
High-intent search queries and answers about installing and using this skill.
Success rate estimation calculates true success rates for production data by applying bias correction. This accounts for known judge inaccuracies, ensuring the estimated success rates reflect actual pipeline performance rather than raw judge outputs.
You measure judge alignment by calculating True Positive Rate (TPR) and True Negative Rate (TNR) against human-labeled data. This allows you to iteratively refine judge prompts to improve evaluation accuracy and reduce false assessments.
You can set up CI evaluation pipelines by deploying validated LLM-as-Judge evaluators. This automates quality assessment within your development lifecycle to continuously catch subjective failure modes before production release.
The best way to evaluate subjective failure modes is using an LLM-as-Judge evaluator. You design specific judge prompts to assess nuanced criteria like empathy or helpfulness, moving beyond simple keyword matching for automated quality assessment.
Success rate estimation calculates true success rates for production data by applying bias correction. This accounts for known judge inaccuracies, ensuring the estimated success rates reflect actual pipeline performance rather than raw judge outputs.
hamelsmuDesign LLM-as-Judge evaluators for subjective criteria with binary pass/fail outcomes.
iani-kuliDesign binary LLM-as-Judge evaluators for subjective failure modes.
dot-promptMap user prompts to prompting frameworks and output structured improvement plans.
boazcstrikeImplement LLM-as-a-Judge evaluation systems with pairwise comparison and bias mitigation.
