skills.rest
enigmaicon-eng
enigmaicon-engCOMMUNITY
Explore all skills by this author →
GitHub

advanced-evaluation

Score or compare LLM outputs with rubric-based LLM-as-judge evaluation and structured JSON results.

1|Updated May 16, 2026
Search Tags:#llm-as-judge#llm evaluation#quality metrics#bias mitigation#pairwise comparison#confidence calibration#rubric generation
One-click install
npx skills add https://github.com/enigmaicon-eng/AI-Enterprise-OS --skill advanced-evaluation-enigmaicon-eng
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: advanced-evaluation
Source: https://github.com/enigmaicon-eng/AI-Enterprise-OS/tree/main/examples/agent-skills/skills/advanced-evaluation
Command: npx skills add https://github.com/enigmaicon-eng/AI-Enterprise-OS --skill advanced-evaluation-enigmaicon-eng

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) and scripts (resource) components.

What problem does it solve?

This Skill helps you reliably judge LLM outputs—especially when using LLM-as-judge—while reducing common evaluation failures like position bias, length bias, and overconfident scoring.

Core Features & Use Cases

  • LLM-as-a-Judge evaluation methods: Direct scoring for objective rubric checks and pairwise comparison for preference-based judgments.
  • Bias mitigation: Position swapping protocols, optional multi-shuffles, anti-length-bias guidance, cross-model evaluation to reduce self-enhancement, and relevance/verbosity controls.
  • Production-grade evaluation design: Confidence calibration, structured JSON outputs, rubric generation to reduce variance, and metric selection for correlation and agreement with humans.
  • Use cases: Selecting the best of two model responses, building automated evaluation pipelines, generating evaluation rubrics, debugging inconsistent judge behavior, and validating correlation between automated and human judgments.

Quick Start

Use the advanced-evaluation skill when you want to compare two model responses using an evaluation rubric and mitigate position bias by swapping response order in two passes.

Frequently Asked Questions about advanced-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate LLM outputs using LLM-as-judge without introducing position bias?▼

Mitigate position bias in LLM-as-judge evaluation by applying position-swap consistency checks, swapping response order across multiple passes to ensure scoring remains stable regardless of input sequence.

What is pairwise comparison for LLM evaluation and when should I use it?▼

Pairwise comparison is an LLM evaluation method for preference-based judgments, used when selecting the best of two model responses by applying rubric-based guidance rather than direct objective scoring.

How do I generate evaluation rubrics to reduce variance in automated scoring?▼

Generate evaluation rubrics to reduce variance by defining specific, evidence-first scoring criteria, guiding the LLM-as-judge to produce consistent, structured JSON results across automated evaluation pipelines.

Can I calibrate confidence scores to validate correlation with human judgments?▼

Yes, calibrate confidence scores in LLM evaluation by applying metric selection logic and correlation reporting to validate agreement between automated judge outputs and human quality assessments.

Why does my LLM judge show length bias when comparing model responses?▼

Length bias occurs when an LLM judge favors longer responses; reduce it by applying anti-length-bias guidance, verbosity controls, and cross-model evaluation to prevent systematic self-enhancement.

Related Skills

Explore More Data & Analytics Skills→
FVossebeldFVossebeld

advanced-evaluation

Automate LLM output evaluation with scoring, pairwise comparisons, and bias mitigation.

Community
Advanced
corticalstackcorticalstack
1

advanced-evaluation

Score and compare LLM outputs with debiased, JSON-formatted evaluations.

Community
Advanced
jiaxiaojunQAQjiaxiaojunQAQ
75

advanced-evaluation

Automate LLM output evaluation with rubric-based judges and bias diagnostics.

Community
Advanced
rohunvorarohunvora
5

advanced-evaluation

Automate LLM-as-a-judge scoring, pairwise comparison, and rubric generation with bias mitigation.

Community
Advanced