skills.rest
goodnight000
goodnight000COMMUNITY
Explore all skills by this author →
GitHub

advanced-evaluation

Evaluate LLM outputs with direct scoring, pairwise comparisons, and rubric generation.

Updated Aug 23, 2026
Search Tags:#LLM#pairwise#rubrics#prompt-design#bias-mitigation#evaluate
One-click install
npx skills add https://github.com/goodnight000/KittyCourt --skill advanced-evaluation-goodnight000
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: advanced-evaluation
Source: https://github.com/goodnight000/KittyCourt/tree/main/.codex/skills/Agent-Skills-for-Context-Engineering-main/skills/advanced-evaluation
Command: npx skills add https://github.com/goodnight000/KittyCourt --skill advanced-evaluation-goodnight000

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides production-grade methodologies for evaluating LLM outputs by using LLMs as judges, enabling consistent quality assurance, bias mitigation, and objective scoring across automated and human-in-the-loop evaluations.

Core Features & Use Cases

  • Direct Scoring: implement structured, criterion-based scoring on objective outputs (e.g., factual accuracy, instruction following).
  • Pairwise Comparison: compare two responses with bias-mitigation strategies such as position swapping and multi-pass evaluation.
  • Rubric Generation: create domain-specific rubrics that define levels, evidence, and edge-case handling to reduce variance.
  • Bias Monitoring: detect and mitigate common evaluation biases (position, length, verbosity, self-enhancement) and calibrate confidence accordingly.
  • Integration: embed evaluation patterns into production pipelines and CI workflows for repeatable quality assessments.

Quick Start

  1. Prepare two model outputs and a clear evaluation prompt, plus a concise rubric describing criteria and weights.
  2. Run a direct scoring pass to rate a single response on objective criteria, then perform a pairwise comparison to choose between alternatives.
  3. Retrieve a structured JSON report containing per-criterion scores, justifications, confidence, and a concise summary.

Frequently Asked Questions about advanced-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I use LLM-as-a-Judge to evaluate model outputs?▼

LLM-as-a-Judge evaluates model outputs by applying structured patterns like direct scoring, pairwise comparisons, and rubric generation to ensure consistent quality assurance and objective scoring across automated pipelines.

What is the best way to reduce bias in pairwise comparison evaluations?▼

Reduce bias in pairwise comparison evaluations by applying position swapping and multi-pass evaluation strategies, which mitigate position and self-enhancement biases while calibrating evaluation confidence.

How do I create domain-specific rubrics for LLM evaluation?▼

Create domain-specific rubrics for LLM evaluation by defining specific levels, required evidence, and edge-case handling parameters to reduce scoring variance and improve evaluation consistency.

Can I integrate LLM evaluation patterns into production pipelines?▼

You can integrate LLM evaluation patterns into production pipelines and CI workflows to automate repeatable quality assessments, generating structured JSON reports with per-criterion scores and justifications.

What do I need to start scoring LLM responses with direct scoring?▼

To start direct scoring, you need well-defined prompts, a concise rubric describing criteria and weights, and the model outputs to evaluate, allowing structured, criterion-based scoring on factual accuracy and instruction following.

Related Skills

Explore More Education & Research Skills→
Sir-chawakornSir-chawakorn
9

llm-eval-harness

Builds an LLM evaluation harness using golden datasets and code-based scorers.

Community
Advanced
pradyumnasagarpradyumnasagar
6

agent-handoff

Automate inter-skill data transfer and provenance capture via JSON Material Passports.

Community
Advanced
khrorekhrore

advanced-evaluation

Automate LLM evaluation pipelines with direct scoring and pairwise comparison.

Community
Advanced
lucasFR2lucasFR2

advanced-evaluation

Automate LLM output evaluation with LLM-as-judge pipelines and bias mitigation.

Community
Advanced